Enterprise information integration processing system based on big data mining

Abstract

The application discloses an enterprise information integrated processing system based on big data mining, and relates to the technical field of big data mining and enterprise information integrated processing.The system comprises a cascaded index construction module, a forgotten request processing module and an integrity repair module.The cascaded index construction module generates blood relationship fingerprints for original records, embeds reverse blood relationship pointers into B+ trees and inverted index leaf nodes, and realizes bidirectional tracing by cooperating with blood relationship Bloom filters and bitmap indexes.The forgotten request processing module selects a difference operator according to an aggregation type, and performs local incremental updating and local rebalancing on affected data units.The integrity repair module generates compensation parameters to maintain statistical smoothness.The system is used for compliance processing of fine-grained data deletion requests in an enterprise big data platform.

Description

Technical Field

[0001] This invention relates to the field of big data mining and enterprise information integration processing technology, specifically to an enterprise information integration processing system based on big data mining. Background Technology

[0002] As enterprises advance their digital transformation, the massive amounts of data they gather through multi-source heterogeneous channels are widely used to build multi-layered materialized views, knowledge graphs, and deep mining models to support business queries and decision analysis. These systems are mostly built on distributed computing frameworks, and the underlying B+ tree index and inverted index are commonly used as the basic structure for storage and retrieval. With the improvement of data protection regulations, users are given the right to request enterprises to delete their personal information, and enterprises are required to remove the relevant information and all derivative traces generated in downstream systems within a reasonable period of time.

[0003] The current standard way to handle deletion requests is as follows: after deleting a specified record in the original data table, since the record has been cleaned and aggregated and written to multiple materialized views and aggregated indexes, the system has difficulty in determining the specific range of downstream data units affected. It can only trigger a full pipeline re-execution for the relevant data domains, rebuild the affected materialized views and recalculate the mining model. After completion, the updated views and models are published for upper-level calls.

[0004] This approach relies on the serial pipeline relationship between the original data table, the integration engine, the view storage layer, and the model training module. Each layer lacks a fine-grained bidirectional mapping relationship. Once deletion is performed, a full recalculation can only be performed from the bottom up, resulting in high computing power consumption, long processing time, and the business system often needs to be shut down and downgraded during the recalculation. The data units after recalculation may also change from the statistical results before deletion, affecting the continuous availability of reports. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing enterprise information integration and processing systems, when performing fine-grained data deletion, lack a fine-grained bidirectional lineage mapping relationship between the original records and downstream materialized views and mined features. This makes it impossible for the system to accurately determine the range of derived data units specifically affected by a record to be deleted. Consequently, a bottom-up, full recalculation approach is used to restore data consistency, resulting in high computational consumption, processing latency that surges with data size, and the need for system shutdown or downgrade during recalculation. Furthermore, recalculated data units may exhibit abrupt changes compared to pre-deletion statistical results, affecting the continuous availability of downstream reports. The goal is to meet the compliance requirements for fine-grained data deletion while avoiding large-scale full recalculation and maintaining the overall smoothness of downstream statistical results after deletion.

[0006] To address the aforementioned technical problems, this invention provides an enterprise information integration and processing system based on big data mining. This system includes a data access and integration engine, a multi-layered materialized view and mining feature storage module, a unified query service layer, a lineage-aware cascaded index construction module, a forgetting request processing module, and a historical smoothing and integrity restoration module.

[0007] Among them, the bloodline-aware cascaded index building module generates a bloodline fingerprint for each original record and writes the bloodline fingerprint into the bloodline Bloom filter and the bloodline bitmap index;

[0008] For each derived data unit, the set of reverse lineage pointers corresponding to the original record set it depends on is recorded, and the set of reverse lineage pointers is stored in the extended field of the corresponding B+ leaf node in the form of a compressed bitmap. At the same time, the corresponding lineage fingerprint identifier is appended to the end of the record list of the inverted index, forming a cascaded index structure that is bidirectionally accessible by forward and reverse mapping, so as to realize fast source tracing and positioning from any original record to all its derived data units.

[0009] Upon receiving a compliant deletion request, the forgetting request processing module parses the set of original record identifiers to be deleted, calculates the corresponding lineage fingerprint, performs member detection through a lineage Bloom filter, locates the set of affected derived data units in the lineage bitmap index, selects the corresponding inverse difference operator according to the aggregation type of each derived data unit, and performs local incremental update and local B+ tree path rebalancing, thereby completing the compliant deletion with a computational complexity far lower than full recalculation.

[0010] The historical smoothing and integrity repair module solidifies the non-sensitive distribution characteristics of the aggregation context of each derived data unit during the index building stage. After the forgetting process is completed, it calculates the compensation coefficient based on the sample size before and after the update and the distribution characteristics, and generates placeholder compensation values ​​to fill the affected dimension components. This maintains the mathematical smoothness of the downstream statistical results while ensuring that the deleted data is physically unrecoverable.

[0011] The present invention also provides a fine-grained data forgetting processing method adapted to the above system. Through steps such as lineage fingerprint detection, affected data unit location, reverse differential update, and local index rebalancing, it achieves accurate tracing and reversible incremental removal of original records in the enterprise information integration processing system.

[0012] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0013] 1. This invention achieves rapid source tracing and location of any original record in its complex derivative chain by natively embedding reverse lineage pointers in the leaf nodes of the B+ tree and inverted index, and by combining lineage Bloom filter and lineage bitmap index. This overcomes the shortcomings of the prior art in which the original table and materialized view are independent of each other and cannot accurately determine the scope of the deletion impact.

[0014] 2. This invention reduces the computational complexity of compliant deletion operations from full recalculation to the order of magnitude related to the logarithm of the number of affected data units and the total size of index records by selecting the corresponding inverse difference operator according to the aggregation type and performing local rebalancing only on the affected B+ tree paths. This significantly reduces the computational power consumption and processing latency of fine-grained data deletion.

[0015] 3. Because the forgetting process in this invention only performs incremental rebalancing on the affected local index paths, there is no need to rebuild the overall materialized view and mining model, which avoids long-term system downtime or degraded operation and improves business continuity.

[0016] 4. This invention maintains the mathematical smoothness and usability of downstream query and statistical report results by solidifying non-sensitive macro-distribution characteristics during the index building stage and adaptively generating compensation parameters after data removal, while ensuring that deleted data is physically unrecoverable. Attached Figure Description

[0017] Figure 1 This is an overall architecture diagram of the system in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the LACI index node structure in an embodiment of the present invention;

[0019] Figure 3 This is a flowchart of fine-grained forgetting processing in an embodiment of the present invention. Detailed Implementation

[0020] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0021] The technical solution of the present invention will be further described below with reference to specific embodiments.

[0022] In this embodiment, the overall architecture of the enterprise information integration and processing system, from bottom to top, includes a multi-source heterogeneous data source layer, a data access and integration engine, a lineage-aware cascaded index construction module, a multi-layer materialized view and mining feature storage module, a forgetting request processing module, a historical smoothing and integrity repair module, and a unified query service layer. The data access and integration engine collects raw records from multi-source heterogeneous data sources such as ERP systems, CRM systems, and external API interfaces, performs cleaning, transformation, and aggregation processing on the raw records, and generates materialized views and deep mining features. While the integration engine processes the raw records, the lineage-aware cascaded index construction module generates a lineage fingerprint for each raw record, and embeds the reverse lineage pointers corresponding to the raw records on which each derived data unit depends into the index structure during the generation of materialized views and mining features. The forgetting request processing module receives external compliant deletion requests, calls the lineage Bloom filter and lineage bitmap index to complete the location of the affected data units, and performs local incremental updates. The historical smoothing and integrity repair module interacts with the forgetting request processing module, reads the solidified context distribution features, generates compensation parameters, and writes them back to the index structure. The unified query service layer provides integrated query and decision support interfaces to the business system.

[0023] In this embodiment, the LACI index structure is a low-level fusion and transformation based on the traditional B+ tree and inverted index. The leaf nodes of the B+ tree are expanded with a reverse lineage pointer area and a context distribution feature area on the basis of the original key value area and data value area. The key value area stores the primary key of the derived data unit.

[0024] The original data value area stores business data such as the aggregation result or feature vector corresponding to the derived data unit;

[0025] The reverse lineage pointer area stores the set of lineage fingerprints of all original records on which the derived data unit depends in the form of a compressed bitmap;

[0026] The context distribution feature area stores non-sensitive macro-distribution statistics of the aggregation context in which the derived data unit is located, which can be called by the historical smoothing and integrity repair module. The lineage Bloom filter and lineage bitmap index serve as a fast index layer for the lineage pointers of leaf nodes, and are placed between the leaf nodes and the original record lineage fingerprint table. The reverse lineage pointer area of ​​the leaf node can be mapped to the original record lineage fingerprint table, and the original record lineage fingerprint table can also be mapped forward to the multiple leaf nodes in which it was generated, forming a bidirectional reachable cascading index feature.

[0027] Regarding the specific implementation of lineage fingerprint embedding and index construction, the data integration engine collects a set of original records from multi-source heterogeneous data sources, assigns a globally unique record identifier to each original record, and calculates its lineage fingerprint for each original record according to the following formula:

[0028] ;

[0029] in, Indicates the first Bloodline fingerprints from the original records;

[0030] Represents a collision-resistant hash function;

[0031] This represents a globally unique identifier for the original record;

[0032] This indicates the timestamp when the original record entered the system;

[0033] This indicates the data source identifier of the original record;

[0034] symbol This indicates a byte-level concatenation operation.

[0035] The calculated kinship fingerprint is inserted into a kinship Bloom filter for subsequent rapid probabilistic determination of the existence of a specific kinship in the system. Simultaneously, the mapping relationship between kinship fingerprints and record identifiers is written into a kinship bitmap index organized by data partitions. The data integration engine performs cleaning, aggregation, and feature processing on the original records according to predefined rules, generating multi-layered materialized views and deep-mining features. For each derived data unit, the set of original records it depends on during construction is recorded, and the corresponding set of reverse kinship pointers is stored in the extended field of the corresponding leaf node in the B+ tree index of that derived data unit in the form of a compressed bitmap. At the same time, the corresponding kinship fingerprint identifier is appended to the end of the record list of the inverted index. The above processing is performed iteratively on all materialized views and mined feature spaces in the system until the index embedding is complete, outputting the final cascaded index structure for subsequent query, tracing, and forgetting processing modules to call.

[0036] To control the storage overhead caused by lineage pointers, the false positive rate of the lineage Bloom filter is estimated by the following formula:

[0037] ;

[0038] in, This indicates the false positive rate of the bloodline Bloom filter;

[0039] This indicates the number of hash functions used in the lineage Bloom filter;

[0040] This indicates the number of bloodline fingerprint elements that have been inserted into the bloodline Bloom filter;

[0041] The bit array length of the bloodline Bloom filter is represented in this embodiment by appropriate selection. and The value of can control the additional storage overhead caused by the lineage pointer to within five percent of the original index size.

[0042] Regarding the handling of fine-grained data forgetting, the forgetting request processing module receives a compliant deletion request, parses it to obtain the set of original record identifiers to be deleted, calculates the lineage fingerprint of each identifier in the set according to the aforementioned formula, and performs member detection in the lineage Bloom filter.

[0043] For a detected bloodline fingerprint, locate its position in the leaf node of the cascading index structure in the bloodline bitmap index to obtain the set of affected derived data units.

[0044] For each derived data unit in the affected derived data unit set, select the corresponding inverse difference operator according to its aggregation type. For summation-type aggregations, update the aggregation value according to the following formula:

[0045] ;

[0046] in, This represents the aggregate total value before the update;

[0047] This represents the updated aggregate total.

[0048] This represents the set of original records that the data unit depended on and that were deleted.

[0049] Representing records The original contribution function for this aggregate value.

[0050] For mean-based aggregations, the mean is recalculated using the following formula:

[0051] ;

[0052] in, This represents the mean before the update;

[0053] This represents the updated mean;

[0054] Indicates the sample size before the update;

[0055] Representing records The corresponding value;

[0056] This indicates the number of records that were deleted.

[0057] For variance-based aggregation, the second-order central moment accumulation is recalculated using the following formula:

[0058] ;

[0059] in, This represents the cumulative amount of the second-order central moments before the update;

[0060] This represents the updated sum of second-order central moments, and the variance can be expressed as... Further calculations show that for counting aggregations, the count value before the update is directly subtracted from the count value. Get the updated count value;

[0061] For the feature vector mining results, update the components of each dimension of the feature vector according to the weighted subtraction and renormalization method.

[0062] After the update result is written back to the leaf node corresponding to the cascaded index structure, the lineage fingerprints related to the record to be deleted are cleared from the reverse lineage pointer set of that leaf node, and the corresponding lineage fingerprints are marked as invalid in the lineage Bloom filter and lineage bitmap index. A local rebalancing operation is performed on the updated leaf node, and only the affected B+ tree paths are split or merged as necessary. No read or write operations are performed on the remaining unaffected nodes. In this embodiment, the overall computational complexity of this forgetting process is:

[0063] ;

[0064] in, This indicates the time complexity of the reverse differential ablation process;

[0065] Indicates the number of affected derived data units;

[0066] This indicates the total size of records in the cascading index structure.

[0067] To facilitate understanding, an example is given to illustrate the specific application of the inverse differential ablation algorithm in this embodiment. Suppose that the aggregated monthly sales value of a region in a materialized view is... The value before the update was 500,000 yuan. The sum of the contributions of the original records involved in this deletion request to the aggregate value is 1,500 yuan. Based on the summation-type difference operator, the updated aggregate value is... Yuan, and let the average order amount of users be a certain feature discovered. The price before the update was 50 yuan, corresponding to the sample size. The value is 1000. The sum of the values ​​corresponding to the deleted records is 150 yuan. The number of deleted records is 4. Based on the mean-type difference operator, the updated mean is 1000. For example, if the active user count in a certain statistical view was 20,000 before the update, and the number of records involved in this deletion is 1, then according to the counting differential operator, the updated count is 19,999. The above update process only involves local numerical calculations and rebalancing of the corresponding index paths, without the need to recalculate the overall view or model.

[0068] Regarding the specific implementation of historical smoothing and integrity restoration, during the index building phase, the historical smoothing and integrity restoration module synchronously extracts the macroscopic non-sensitive distribution features of the data on which each materialized view or mined feature depends, including category proportions, numerical range distributions, and time series trends, and stores them permanently in the metadata area corresponding to the leaf node of the cascading index structure for that data unit. When the forget request processing module completes the update of a certain data unit, it reads its permanently stored contextual distribution features and the sample size before the update. Compared with the updated sample size Based on the proportion of samples of each non-sensitive category before deletion and the proportion of remaining samples after deletion recorded in the context distribution features, the adaptive compensation coefficient is calculated:

[0069] ;

[0070] in, Indicate category The adaptive compensation coefficient;

[0071] Indicate category The number of samples before the update;

[0072] Indicate category The number of remaining samples after the update;

[0073] This represents the total sample size before the update;

[0074] This represents the updated total sample size.

[0075] For example, let's say the number of samples before the update for a certain age group category. The number is 3000; this represents the number of remaining samples in this category after the update. The total sample size before the update was 2996. The updated total sample size is 20,000. The value is 19996. Based on the above formula, the compensation coefficient corresponding to this category is calculated. .

[0076] The placeholder compensation value is generated based on the compensation coefficient and filled into the dimension component of the affected aggregation result or feature vector in the corresponding category. This ensures that the updated statistics are consistent with the macro distribution trend before deletion in terms of mathematical expectation. The placeholder itself is calculated only from the fixed macro distribution characteristics and the statistics of the remaining samples and does not contain any specific numerical characteristics of the deleted records. This ensures that sensitive fine-grained data cannot be physically reversed and reconstructed. Consistency verification is performed on the compensated data units to compare whether the deviation of downstream key report indicators before and after the repair is within the preset tolerance threshold.

[0077] If the offset exceeds the tolerance threshold, the compensation coefficient will be adjusted and recalculated until the tolerance requirement is met. The final confirmed compensation parameters and placeholders will be written into the metadata field of the corresponding leaf node of the cascaded index structure, and the temporary computing resources occupied by this forget request will be released.

[0078] In another embodiment of the present invention, the storage implementation of the lineage mapping relationship can also be implemented without using the combination of lineage Bloom filter and lineage bitmap index. Instead, a lightweight graph database can be used to maintain the lineage edge relationship between the original record and the derived data unit separately. The location of the affected data unit is completed by graph traversal query. This method has a certain degree of scalability in scenarios where there are multi-level nested dependencies in the lineage relationship. In yet another embodiment of the present invention, the implementation of fine-grained forgetting can also be implemented without using the method of selecting the difference operator by type. Instead, a local recalculation window is set for the affected data unit, and only a small-scale recalculation is performed on the affected materialized view partition or local substructure of the mining model. This method is suitable for some complex mining model scenarios where it is difficult to express the difference relationship using algebraic operations. The above embodiments do not depart from the technical concept of the present invention and are alternative solutions that can be implemented by the present invention.

[0079] It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. All equivalent substitutions or improvements made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

[0080] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. An enterprise information integration and processing system based on big data mining, comprising a data access and integration engine, a multi-layer materialized view and mining feature storage module, and a unified query service layer, wherein the data access and integration engine is used to collect raw records from multi-source heterogeneous data sources, and perform cleaning and aggregation processing on the raw records to generate derived data units stored in the multi-layer materialized view and mining feature storage module, characterized in that, Also includes: The lineage-aware cascaded index construction module is used to generate a lineage fingerprint for each original record collected by the data access and integration engine, and write the lineage fingerprint into the lineage Bloom filter and the lineage bitmap index. The lineage-aware cascaded index construction module is also used to record the set of reverse lineage pointers corresponding to the set of original records that the derived data unit depends on for each derived data unit, store the set of reverse lineage pointers in the form of a compressed bitmap in the extended field of the corresponding leaf node in the B+ tree index of the derived data unit, and append the corresponding lineage fingerprint identifier to the end of the record list of the inverted index, forming a cascaded index structure that is bidirectionally accessible by forward and reverse mapping. The cascaded index structure makes it possible to directly hit the corresponding leaf node for tracing and locating any original record identifier with only one Bloom filter member detection and one bitmap index lookup, without having to perform multi-level traversal of the lineage graph structure. This reduces the time complexity of tracing and locating from the order of magnitude related to the lineage link depth of the existing graph traversal scheme to the constant level. The forgetting request processing module is used to receive compliant deletion requests, parse the set of original record identifiers to be deleted, calculate the corresponding lineage fingerprint for each identifier in the identifier set, perform member detection in the lineage Bloom filter, locate the leaf node position in the cascaded index structure in the lineage bitmap index for the detected lineage fingerprint, and obtain the set of affected derived data units. The forgetting request processing module is also used to select the corresponding inverse difference operator according to the aggregation type of each derived data unit in the set of affected derived data units, calculate the update result of the derived data unit after removing the original record to be deleted, write the update result back to the corresponding leaf node in the cascaded index structure, clear the lineage fingerprint corresponding to the original record to be deleted in the inverse lineage pointer set of the leaf node, and perform a local rebalancing operation on the affected B+ tree path. The historical smoothing and integrity repair module is used to read the fixed context distribution characteristics of the derived data unit and the sample capacity before and after the update after the forgetting request processing module completes the update of the derived data unit, calculate the compensation coefficient according to the context distribution characteristics, generate placeholder compensation values, and fill the placeholder compensation values ​​into the affected dimension components of the derived data unit.

2. The system according to claim 1, characterized in that, The B+ leaf nodes in the cascading index structure include a key value area, an original data value area, a reverse lineage pointer area, and a context distribution feature area. The key value area is used to store the primary key of the derived data unit. The original data value area is used to store the business data corresponding to the derived data unit. The business data includes the aggregation result and feature vector. The reverse lineage pointer area is used to store the lineage fingerprint set of the original records on which the derived data unit depends in the form of a compressed bitmap. The context distribution feature area is used to store the non-sensitive distribution statistics of the aggregation context in which the derived data unit is located.

3. The system according to claim 1, characterized in that, The bloodline-aware cascaded index construction module calculates the anti-collision hash value by concatenating the globally unique identifier, the timestamp of entering the system, and the data source identifier of each original record, thereby obtaining the bloodline fingerprint corresponding to the original record.

4. The system according to claim 1, characterized in that, The inverse difference operator includes a summation-type difference operator, a mean-type difference operator, a variance-type difference operator, a counting-type difference operator, and an eigenvector-type difference operator. The summation-type difference operator is used to subtract the sum of contributions of the original records to be deleted from the aggregated total value before the update to obtain the updated aggregated total value. The mean-type difference operator is used to calculate the updated mean based on the sample size before the update, the mean before the update, and the corresponding values ​​of the original records to be deleted. The variance-type difference operator is used to calculate the updated accumulated second-order central moments based on the accumulated second-order central moments before the update and the corresponding values ​​of the original records to be deleted. The counting-type difference operator is used to subtract the number of original records to be deleted from the count value before the update to obtain the updated count value. The eigenvector-type difference operator is used to perform weighted subtraction and renormalization processing on each dimension component of the eigenvector.

5. The system according to claim 1, characterized in that, When the forget request processing module performs a local rebalancing operation on the affected B+ tree path, it only performs structural adjustments on the nodes on the path from the leaf node to the root node. The structural adjustments include node splitting and node merging, and it does not perform read / write operations on the remaining unaffected nodes.

6. The system according to claim 1, characterized in that, The bloodline Bloom filter is used to probabilistically determine whether a specified bloodline fingerprint exists in the cascaded index structure. The bloodline bitmap index is used to record the mapping relationship between the bloodline fingerprint and its storage location in the cascaded index structure. Organized according to data partitions, each data partition maintains an independent mapping sub-table from bloodline fingerprint to storage location, allowing multiple query requests to read the mapping sub-tables of different data partitions in parallel.

7. The system according to claim 1, characterized in that, The historical smoothing and integrity repair module calculates the compensation coefficient corresponding to the non-sensitive category based on the number of samples of the non-sensitive category before the update, the number of remaining samples of the non-sensitive category after the update, the total sample size before the update, and the total sample size after the update. The placeholder compensation value is calculated by the compensation coefficient and the statistics of the remaining samples after the update, and does not include the numerical characteristics of the original record to be deleted.

8. The system according to claim 7, characterized in that, The historical smoothing and integrity repair module is also used to compare whether the offset of downstream key report indicators before and after filling the placeholder compensation value is within a preset tolerance threshold. When the offset exceeds the preset tolerance threshold, the compensation coefficient is adjusted and the placeholder compensation value is regenerated.

9. A fine-grained data forgetting processing method for an enterprise information integration processing system, applied to the system described in claim 1, characterized in that, include: Receive compliant deletion requests and parse them to obtain the set of original record identifiers to be deleted; For each identifier in the identifier set, calculate the corresponding bloodline fingerprint and perform member detection in the bloodline Bloom filter; For the detected bloodline fingerprint, locate its leaf node position in the cascaded index structure in the bloodline bitmap index to obtain the set of affected derived data units; For each derived data unit in the set of affected derived data units, select the corresponding inverse difference operator according to the aggregation type of the derived data unit, and calculate the update result of the derived data unit after removing the original record to be deleted; The update result is written back to the corresponding leaf node in the cascading index structure, the lineage fingerprint corresponding to the original record to be deleted is cleared from the reverse lineage pointer set of the leaf node, and a local rebalancing operation is performed on the affected B+ tree path.

10. The method according to claim 9, characterized in that, Also includes: Read the fixed context distribution features of the derived data unit and the sample size before and after the update, calculate the compensation coefficient based on the context distribution features, generate placeholder compensation values, and fill the affected dimension components of the derived data unit with the placeholder compensation values.

Images