A method and system for configurable data synchronization in the private equity field

By constructing a business object meta-model and an incremental synchronization strategy, the logical table name and physical table name are decoupled, achieving efficient and reliable data synchronization and indicator calculation in the private equity field. This solves the problems of high repetition and data inconsistency in existing synchronization tools and supports the management of various private equity indicators.

CN123115276APending Publication Date: 2026-11-17ZHILI INFORMATION TECH (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611621463.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies for synchronizing multi-source heterogeneous data in the private equity field suffer from problems such as large workload, high redundancy, poor reliability of incremental synchronization, and data inconsistency. Furthermore, the calculation of indicators becomes distorted after changes in historical data, and the configuration and maintenance costs are high.

Method used

Construct a business object metamodel, including source logical table name, target table information, field mapping relationship, incremental synchronization strategy and statistical indicators. Decouple logical table name and physical table name through data access layer, and achieve idempotent writing and breakpoint resume by incremental identifier field extraction and database log synchronization. Support recalculation of indicator calculation adapter.

Benefits of technology

It reduces development workload, improves the reliability of incremental data, avoids data inconsistency, supports the management of various private equity indicators, and solves the problem of statistical distortion caused by changes in historical data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115276A_ABST
    Figure CN123115276A_ABST
Patent Text Reader

Abstract

This invention discloses a configurable data synchronization method and system in the field of private equity, relating to the field of data processing technology. The method includes: obtaining a business object metamodel based on the synchronization task configuration and generating read / write scripts; writing extracted, cleaned, and transformed business data idempotently to a big data cluster according to the primary key; during incremental log synchronization, connecting a full snapshot with log positions to write events involving changes in indicator calculation fields or business date fields to a change event storage area; calling an indicator calculation adapter to generate initial statistical results; determining the affected time window and related indicators based on unprocessed events; recalculating and replacing the original indicator values ​​according to the time range to which the related indicators depend, forming the final statistical results and idempotently writing them back to the business system database; and ensuring that the statistical results correspond to the synchronized business data through change location and recalculation result replacement.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a configurable data synchronization method and system in the field of private equity. Background Technology

[0002] In the private equity sector, business objects such as funds, limited partners, projects, fundraising ledgers, capital contribution ledgers, investment ledgers, and exit ledgers are stored in different business system databases. Currently, synchronizing this multi-source heterogeneous data typically requires developers to write separate data extraction, cleaning, transformation, and loading programs for each source logical table and target data table. For example, when adding a new fund capital contribution ledger synchronization task, code for data extraction, field mapping, cleaning, and loading from the source capital contribution ledger table to the target data table needs to be written manually; similar code needs to be rewritten when synchronizing a new project investment ledger, resulting in a large workload and high repetition.

[0003] While existing configuration-based synchronization tools can generate synchronization scripts through field mapping relationships, their incremental synchronization typically relies on a single identifier field. When logical deletions or inconsistencies in transaction commit times and timestamp order occur in the source business system, incremental capture based on the identifier field can lead to data loss or duplication. Furthermore, when synchronization tasks are loaded onto a big data cluster, the lack of idempotency guarantees and breakpoint resume mechanisms during interruptions can easily lead to data inconsistencies. When writing statistical results back to the business system database, it's also necessary to prevent duplicate writes that could cause inconsistencies. In incremental synchronization scenarios based on database logs, historical data corrections in the private equity sector, such as changes in investment dates or investment amounts, require recalculation of time-window-based statistical indicators. Existing configuration-based synchronization tools cannot automatically identify and trigger recalculation, resulting in distorted statistical results. Additionally, when the source business system undergoes table partitioning or field renaming, the source logical table names and field names in the synchronization configuration need to be modified synchronously, leading to high configuration and maintenance costs.

[0004] Therefore, a technical solution is needed that can achieve multi-object data synchronization in a configurable manner in the private equity field, while taking into account the reliability of incremental capture, unified management of indicator calculation, and cascading recalculation of indicators after changes in historical data. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a configurable data synchronization method and system in the field of private equity, solving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Firstly, a method for synchronizing allocation-based data in the private equity field includes the following steps:

[0008] Construct a business object metamodel and store it in the storage module. The business object metamodel includes the source logical table name, target table information, write-back target table information, field mapping relationship, incremental synchronization strategy, incremental identifier field and statistical indicators of the business object.

[0009] The incremental synchronization strategy is either based on the extraction of the incremental identifier field or based on the incremental synchronization of the database logs. The statistical indicators include: indicator type, indicator calculation adapter identifier, business date field, indicator calculation field and time window rules. The indicator calculation field is the numerical field that affects the indicator calculation result. The business date field independently participates in event filtering and time window positioning.

[0010] Receive synchronization task configuration. The synchronization task configuration includes at least the business object identifier to be synchronized and the synchronization mode, which is either full synchronization or incremental synchronization.

[0011] Retrieve the corresponding business object metamodel from the storage module based on the business object identifier;

[0012] Based on the business object metamodel and synchronization mode, read and write scripts are generated. Before generating the read script, the source logical table name is resolved to the actual physical table name through the data access layer. When the synchronization mode is full synchronization, a full read script is generated. When the synchronization mode is incremental synchronization and the incremental synchronization strategy is based on the extraction of the incremental identifier field, filtering conditions based on the incremental identifier field are generated in the read script. When the synchronization mode is incremental synchronization and the incremental synchronization strategy is based on incremental synchronization of database logs, a read script for executing a full snapshot and log parsing configuration are generated.

[0013] The scripts for reading and writing are executed to extract business data from the source database, clean and transform it according to the field mapping relationship, and then write it to the big data cluster in an idempotent manner according to the primary key.

[0014] When the incremental synchronization strategy is based on database logs, it also includes:

[0015] Before performing a full snapshot, the log position of the source database is recorded; a full snapshot is performed, loading all data from the source database into the big data cluster; after the full snapshot is completed, incremental logs are read starting from the log position, and incremental data is loaded into the big data cluster; when reading incremental logs, if a change event involves a change in any statistical indicator's calculation field or business date field, the change event is written to the change event storage area; wherein, the change event record at least includes the primary key of the change record, a list of changed fields, and the old value and new value of each changed field before the change;

[0016] Based on statistical indicators, the corresponding indicator calculation adapter is invoked to generate statistical tasks and execute them in the big data cluster to obtain the final statistical results. The statistical calculations include: incremental aggregation of business data already loaded in the big data cluster according to time windows to obtain initial statistical results; reading change events with an unprocessed status from the change event storage area; when the change field list in the change event contains the indicator calculation field or business date field of any statistical indicator, determining the affected time window and related indicators according to the time window rules of that statistical indicator; rereading all business data within the affected time window and performing aggregation calculations to obtain recalculation results; replacing the original indicator values ​​of the corresponding time window in the initial statistical results with the recalculation results to obtain the final statistical results; and updating the processing status of processed change events to processed.

[0017] The final statistical results are written back to the business system database in idempotent order based on the primary key.

[0018] As a further aspect of the present invention, the business objects include at least one of funds, limited partners, projects, fundraising ledgers, capital contribution ledgers, investment ledgers, and exit ledgers; the target table information includes the target business system identifier, the target database instance identifier, and the target data table identifier; the write-back target table information includes the write-back target business system identifier, the write-back target database instance identifier, and the write-back target data table identifier; the field mapping relationship includes the correspondence between the source logical table field names and the target data table field names.

[0019] As a further aspect of the present invention, the data access layer resolves the source logical table name to the actual physical table name, including:

[0020] If the source business system has tables, the source logical table name will be resolved into multiple actual physical table names according to the table partitioning rules.

[0021] If there are field renaming issues in the source business system, the source logical table field names in the business object metamodel will be resolved to the actual physical table field names according to the mapping rules.

[0022] As a further aspect of the present invention, the time window rules include time window division units, a list of statistical time ranges for indicators, and recalculation rules;

[0023] The recalculation rules are as follows: When the business date field changes, the time windows corresponding to both the old and new values ​​are determined as the affected time windows; when the business date field remains unchanged but the indicator calculation field changes, the affected time window is determined based on the business date corresponding to the event; indicators whose calculation dependency time range intersects with the affected time window are determined as related indicators, and recalculated according to the complete time range that each related indicator depends on.

[0024] The method for determining intersection is as follows: both the time window and the statistical time range of the indicator are intervals that include the start point but not the end point; the time window is compared with each time range in the statistical time range list one by one. If the start point of the time window is less than the end point of the time range, and the end point of the time window is greater than the start point of the time range, then the two are determined to intersect.

[0025] As a further aspect of the present invention, when the incremental synchronization strategy is based on the extraction of the incremental identifier field, the incremental identifier field is a timestamp field or an auto-incrementing field.

[0026] The timestamp field is applicable to records whose order is consistent with the commit visibility order. The upper bound of the previous successful synchronization range is used as the lower bound of this time, and the upper bound of the range determined at the start of the task is used as the upper bound of this time. The auto-increment field is applicable to records that are only appended and whose commit visibility order is consistent with the auto-increment order. Records that are greater than the maximum value of the previous successful synchronization and do not exceed the upper bound determined in this time are extracted. When the corresponding order conditions are not met or when it is necessary to capture historical updates or deletions, incremental synchronization based on database logs is used.

[0027] As a further aspect of the present invention, when the synchronization mode is incremental synchronization and the incremental synchronization strategy is incremental synchronization based on database logs, the log parsing configuration includes: log source information, subscribed database tables, event filtering rules, and field parsing mapping.

[0028] As a further aspect of the present invention, the indicator calculation adapter is pre-configured with at least one calculation module among fund size, limited partner investment progress, project investment cost, exit return, invested capital dividend rate, total return multiple, and internal rate of return; the indicator calculation adapter includes an internal rate of return calculation module, which calculates the internal rate of return through a discounted cash flow iterative algorithm.

[0029] As a further embodiment of the present invention, the event storage area is changed to a relational database table, a message queue, or a structured log file in a distributed file system;

[0030] The change event log includes the source table and record primary key, a list of changed fields, the old and new values ​​of the changed fields, the business date corresponding to the event, the task batch number, and the processing status. The initial processing status is unprocessed. The task batch number is generated when the synchronization task starts and is used during retries. Checkpoints where data loading and event persistence are both successful are recorded. During retries, the part confirmed at this checkpoint is skipped, and the part that has not yet been confirmed is re-executed.

[0031] After the corresponding final statistical results are persisted and successfully written back, the processing status of the corresponding change event will be updated to "processed".

[0032] Secondly, a configuration-based data synchronization system for the private equity sector, the system being used to execute a configuration-based data synchronization method for the private equity sector, the system comprising:

[0033] The storage module is used to store the meta-model of business objects;

[0034] The receiving module is used to receive the synchronization task configuration.

[0035] The acquisition module is used to obtain the corresponding business object metamodel based on the business object identifier;

[0036] The generation module is used to generate read scripts and write scripts based on the business object metamodel and synchronization mode, and resolves the source logical table name through the data access layer before generating the read script;

[0037] The execution module is used to execute read and write scripts, extract, clean, transform and load data, and record log positions, read incremental logs and write to the change event storage area during incremental synchronization based on database logs.

[0038] The statistics module is used to call the indicator calculation adapter to generate and execute statistical tasks based on the statistical indicators in the business object metamodel, obtain the initial statistical results, determine the affected time window based on the unprocessed change events in the change event storage area and perform recalculation to obtain the final statistical results, and update the change event processing status.

[0039] The write-back module is used to write the final statistical results back to the business system database in an idempotent manner according to the primary key.

[0040] Thirdly, a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a configurable data synchronization method in the field of private equity.

[0041] This invention provides a method and system for configurable data synchronization in the private equity field. Compared with existing technologies, it has the following advantages:

[0042] By constructing a business object metamodel, the source logical table name, target table information, write-back target table information, field mapping relationships, incremental synchronization strategy, incremental identifier field, and statistical indicators are uniformly abstracted into machine-readable structured metadata. The source logical table name is decoupled from the actual physical table name through the data access layer, so that adding a new synchronization task only requires specifying the business object identifier and synchronization mode, without having to repeatedly write data extraction, cleaning, transformation, and loading programs, thus reducing the workload of development.

[0043] This is based on incremental identifier field extraction and database log synchronization. The latter records log positions before a full snapshot and reads incremental logs from those positions after the snapshot, reducing incremental data loss and duplication and improving reliability.

[0044] By using idempotent writes by primary key, managing task batch numbers and checkpoint information, the system enables breakpoint resumption and retrying of synchronous tasks. Furthermore, by using idempotent writes by primary key, the statistical results are written back to the business system database to avoid data inconsistency caused by synchronization interruptions.

[0045] By using an indicator calculation adapter, the calculation logic of statistical indicators is decoupled from the business object meta-model, supporting the management of various private equity indicators such as fund size, limited partner investment progress, project investment cost, exit returns, invested capital dividend rate, total return multiple, and internal rate of return.

[0046] During incremental synchronization based on database logs, the system writes change events involving changes in indicator calculation fields or business date fields to the change event storage area and records the processing status and task batch number. The statistics module reads unprocessed change events from the change event storage area, determines the affected time window and related indicators according to the time window rules, and performs recalculation to solve the problem of statistical distortion caused by historical data correction in private equity business. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the method of the present invention.

[0048] Figure 2 This is a system block diagram of the system of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Please see Figure 1 and Figure 2 The present invention provides the following technical solutions:

[0051] Example 1

[0052] This invention relates to a method for configurable data synchronization in the private equity field, comprising the following steps:

[0053] S1: Construct a business object metamodel and store it in the storage module. The business object metamodel is a structured metadata set used to describe the core configuration information required for the synchronization and indicator statistics of various business objects in the private equity field. Business objects include funds, limited partners, projects, fundraising ledgers, capital contribution ledgers, investment ledgers, exit ledgers, etc.; and each business object corresponds to one business object metamodel.

[0054] The business object metamodel includes business object identifier, source logical table name, target table information, write-back target table information, field mapping relationship, incremental synchronization strategy, incremental identifier field, and statistical indicators;

[0055] The source logical table name is the logical table name configured in the business object metamodel to identify the source data. It is used by the data access layer to resolve to the actual physical table name before the read script is generated. The actual physical table name is the real physical table name in the source database after resolution. When the source table has table partitioning or field renaming, the data access layer resolves the source logical table name to one or more actual physical table names and resolves the source logical table field names to actual physical table field names. The target table information is used to locate the target data table when loading data into the big data cluster, including the target business system identifier, target database instance identifier, and target data table identifier. The write-back target table information is used to write the final statistical results back to the business system database, including the write-back target business system identifier, write-back target database instance identifier, and write-back target data table identifier. The field mapping relationship is the correspondence between the source logical table field names and the target data table field names. The incremental synchronization strategy specifies the incremental synchronization method applicable to this business object, with a value based on the incremental identifier field. The extraction can be based on incremental data extraction or incremental synchronization based on database logs. If the extraction is based on an incremental identifier field, then the incremental identifier field must be specified. If the incremental synchronization is based on database logs, then it is not necessary to rely on the incremental identifier field on the business table. Incremental identifier field: When the incremental synchronization strategy is based on the incremental identifier field, specify the field used to identify incremental data, such as a timestamp field or an auto-increment field. Statistical indicators include indicator type, corresponding indicator calculation adapter identifier, indicator calculation field, business date field, and time window rules. Time window rules are a set of rules used to determine the time windows affected by the calculation and recalculation of statistical indicators. They include time window division units, indicator statistical time range list, and recalculation rules. In this embodiment, for time window division units, such as by day, by month, by quarter, etc., the indicator statistical time range list is a set of time ranges manually configured for indicator calculation, specifically describing the time ranges that each statistical indicator depends on when performing statistical calculations. Each time range includes start time and end time.

[0056] The recalculation rules are as follows: When the business date field changes, the time windows corresponding to both the old and new values ​​are determined as the affected time windows; when the business date field remains unchanged but the indicator calculation field changes, the affected time window is determined based on the business date corresponding to the event; indicators whose calculation dependency time range intersects with the affected time window are determined as related indicators, and recalculated according to the complete time range that each related indicator depends on.

[0057] The method for determining intersection is as follows: both the time window and the statistical time range of the indicator are intervals that include the start point but not the end point; the time window is compared with each time range in the statistical time range list one by one. If the start point of the time window is less than the end point of the time range, and the end point of the time window is greater than the start point of the time range, then the two are determined to intersect.

[0058] S2: Receive synchronization task configuration. The synchronization task configuration includes at least the business object identifier to be synchronized and the synchronization mode, wherein the synchronization mode is full synchronization or incremental synchronization.

[0059] S3: Retrieve the corresponding business object metamodel from the storage module based on the business object identifier.

[0060] S4: Generate read and write scripts based on the business object metamodel and synchronization mode.

[0061] The script is generated in the following way:

[0062] Load the corresponding reader template from the template library based on the source database type;

[0063] Load the corresponding writer template from the template library based on the target database type;

[0064] Before generating the read script, the data access layer resolves the source logical table name to the actual physical table name. The data access layer maintains the mapping rules from the source logical table name to the actual physical table name. If the source business system has table partitioning, the source logical table name is resolved to multiple actual physical table names according to the partitioning rules. If the source business system has field renaming, the source logical table field name in the business object metamodel is resolved to the actual physical table field name according to the mapping rules. The mapping rules of the data access layer can be stored in an independent configuration table or a configuration file in a distributed configuration center. The mapping rule configuration items include: logical table name, actual physical table name, partitioning rules (such as partitioning by month or by tenant), and field renaming mapping relationships. The data access layer provides a mapping rule maintenance interface, allowing operations and maintenance personnel to add, modify, and delete mapping rules through the management interface or configuration update interface. When the source business system undergoes table partitioning or field renaming, only the mapping rules of the data access layer need to be updated; the business object metamodel and the synchronous script generation logic do not need to be changed.

[0065] For example, when partitioning tables by month, the data access layer reads the list of months and physical table names corresponding to the logical tables from the configuration table, selects the physical tables with overlapping months according to the current read time range, and enumerates all registered physical tables for a full read without time range restrictions; when partitioning tables by tenant, the physical tables in the configuration are selected according to the tenant identifier of the current synchronization; for each physical table, the field names are replaced by field renaming mappings, and the result columns are restored to logical field names and written according to the field mapping relationship. If no table or necessary field mapping is found, the task is terminated and the configuration item is recorded; this process ensures that changes to physical tables are limited to the access layer configuration and do not change the logical field correspondence.

[0066] The parsed actual physical table name is filled into the reader template to generate the source database connection and query statement; the target table information is filled into the writer template to generate the target data table connection and insert statement; the field mapping relationship is filled into the writer template to generate the field selection list and field correspondence; after reading, the amount text is parsed into a fixed-point number according to the decimal places of the source field, the date text is parsed into a business date according to the format of the source field, and the valid value is converted into the target field type; if the primary key is missing, the amount is invalid, or a necessary field is missing, an error is recorded and the batch is stopped, and the checkpoint is not advanced; different currencies are not directly summed, and amount aggregation is only performed when the input is already in the same currency; since the source of the exchange rate for raw materials is not given, no separate exchange rules are set.

[0067] When the synchronization mode is full synchronization, a full read script is generated, but incremental filtering conditions and log parsing configurations are not generated.

[0068] When the synchronization mode is incremental synchronization and the incremental synchronization strategy is based on the extraction of the incremental identifier field, the upper bound of this task is determined before the task starts reading. The timestamp field uses the condition that it is greater than the previous successful range upper bound and less than or equal to the current upper bound, and the auto-increment field uses the condition that it is greater than the previous successful maximum value and less than or equal to the current upper bound. The upper bound is persisted only after all data within the range has been successfully written. Pagination uses a stable sort composed of the incremental identifier field and the primary key to distinguish records with the same timestamp. In case of failure, the unconfirmed range is reread and overwritten. Both the timestamp and auto-increment fields must meet the applicable condition that the submission visibility order is consistent with the field order. The auto-increment mode is only used for appending data. If the condition is not met or there are historical updates or deletions, the log incremental strategy is used. Since the next lower bound comes from the completed reading range rather than the task completion time, the range that has not been read between the end of this reading and the end of the task will not be skipped.

[0069] When the synchronization mode is incremental synchronization and the incremental synchronization strategy is incremental synchronization based on database logs, a read script for performing a full snapshot and a log parsing configuration are generated during initialization. The log parsing configuration includes log source information, subscribed database tables, event filtering rules, and field parsing mappings. The system calls the change data capture component to subscribe to the source database logs.

[0070] Meanwhile, when generating the write script, the target record's identity is determined by the source table and the record's primary key; when multiple source databases or partitioned tables have the same primary key, the target key also includes the source system or partitioned table identifier to keep different records distinguishable; if a record does not exist, it is inserted; if a record exists, it is updated with the complete field value of this update; deletion events delete the target record using the same key or write a deletion mark and exclude it in the statistical query; duplicate deletions of non-existent records are considered as the deletion operation being completed; the same source is processed according to the log submission order to ensure that older content does not overwrite the target record after newer content.

[0071] S5: Executes read and write scripts to extract business data from the source database, cleans and transforms it according to field mapping relationships, and writes it to the big data cluster idempotently according to the primary key; the big data cluster is a distributed data storage and computing cluster used for storing and computing synchronized data.

[0072] A task batch number is generated at the start of a new task, and the original batch number is used for retries in case of failure. The checkpoint table records the batch number, synchronization stage, range of successfully completed data, and log points that have been successfully applied. Log points are only advanced after the corresponding data has been persisted and the required change events have been persisted. Data and checkpoints are committed together when they are in the same transaction resource. When they belong to different resources, data and events are committed first, followed by checkpoints. It cannot be assumed that independent databases and clusters naturally share transactions. If any step fails, checkpoints in the incomplete range are not advanced. During recovery, the process is replayed from the last successful range. Repeated execution is prevented from accumulating by relying on record overwriting and event deduplication. Batches of the same business object are advanced in the order of confirmation. The results of the next batch are not published until the previous batch has been written back.

[0073] During the initial incremental log synchronization, the change data capture component obtains corresponding consistent snapshots and log start points, and retains logs starting from that point. A full read of the same read view using that snapshot is performed. After the snapshot is loaded, logs after that point are applied in the commit order. During initialization, statistical results that are not yet fully connected are not published. Reading continues directly when a successful checkpoint exists; if the snapshot expires or the required logs are missing, reading stops and re-initialization occurs. Missing intervals cannot be marked as complete. The execution module uses the union of each metric calculation field and the business date field as the event filtering field set. Any change in a dependent field persists the event. Logical deletion of fields or physical deletion involving records already included in the statistics also retains the event and the date before deletion. Each event saves the source table, record primary key, old and new values ​​of the changed fields, the corresponding business date, log position and event order, task batch number, and initial unprocessed state. Because the date field participates in filtering independently, correcting only the funding date can trigger processing in both the original and new time windows.

[0074] When using relational database tables, a unique key for each event is formed by the source system, source table, log location, and the order of events within the same location. This key stores the old value, new value, business date, batch number, and processing status. Duplicate events are skipped based on the unique key, and updates from different times are not merged due to identical primary keys. When using message queues, events carry the same identity information. Consumers acknowledge the event after the final statistical results are persisted and successfully written back. If the result fails or is not acknowledged, it is resubmitted and deduplicated according to the event identity. When using structured log files, each line stores a complete event. An independent status table stores the unprocessed or processed status using the file identifier and record sequence number. Incomplete records at the end of the file are not acknowledged and are rewritten by the production end. Because the processing completion status corresponds to the identity of the complete event, two consecutive changes to the same record will not be mistakenly treated as one.

[0075] The statistical processing uses the data range confirmed by the checkpoint as the input boundary. A specific execution sequence is as follows: load the current batch of business data, persist the corresponding event, submit the successful checkpoint, perform statistical recalculation, persist the result, write back and confirm the event, and pause the loading of subsequent batches of the same business object before the end of the current batch of statistical reading. In this way, the date used by the event and the data read by the recalculation belong to the same confirmed state, avoiding the omission of the original window due to subsequent batch changes during the event processing. This sequence can be implemented by the synchronous task execution module to schedule each stage in sequence.

[0076] S6: Based on the statistical indicators in the business object metamodel, call the corresponding indicator calculation adapter to generate statistical tasks and execute them in the big data cluster to obtain the final statistical results.

[0077] The statistical indicators in the business object metamodel specify the indicator type and indicator calculation adapter identifier. The system matches the corresponding calculation module from a pre-built indicator calculation adapter library based on the statistical indicators. In this embodiment, the indicator calculation adapter is a pluggable software component that encapsulates the corresponding calculation logic; the pluggability is achieved through the Java SPI (Service Provider Interface) mechanism: each indicator calculation adapter implements a unified adapter interface, which includes an indicator type identifier, calculation logic entry method, etc.; at system startup, it is automatically registered to the adapter registry through the SPI configuration file. When statistical calculations need to be performed, the system searches for the corresponding adapter instance in the registry based on the indicator calculation adapter identifier in the statistical indicator metamodel, calls its calculation logic to generate a statistical task, and executes it. For example, the fund size calculation module sums the investment amount field in all investment records under the same fund; the indicator calculation adapter includes an internal rate of return (IRR) calculation module, which calculates the IRR using a discounted cash flow iterative algorithm.

[0078] When registering an adapter, the uniqueness of the adapter identifier is verified. The statistics module passes the business object identifier, statistical range, field mapping, and confirmed data range to the entry method. The adapter returns the results organized by business object, indicator type, and statistical range identifier, as well as the valid or abnormal status. If the configured adapter does not exist, the fields are insufficient, or the parameters do not meet the calculation conditions, the corresponding indicator task is terminated and unprocessed events are retained, without outputting default values. Since the execution entry point clearly defines the input, output, and failure status, retries will not mistakenly mark missing calculations as successes.

[0079] Statistical calculations specifically include the following sub-steps:

[0080] S6-1: Incremental aggregation yields initial statistical results;

[0081] During the initial snapshot statistics, an aggregated intermediate quantity is established from zero for the same business object and statistical range. A full reconstruction, before release, replaces the target business object's data set with the source snapshot record set, removing records from the target that are no longer in the source set. Subsequent new records are categorized into corresponding windows based on business date; monetary indicators accumulate the corresponding amount, proportional indicators update the incremental numerator, and the ratio is calculated by reading the denominator at the corresponding statistical time point. For the internal rate of return (IRR) indicator, new records are incorporated into the cash flow series, and the discounting process is called, without adding the proportional value or the IRR itself. New records can change the statistical values ​​of existing windows, but... The correction of old values ​​caused by new or deleted data is not directly added at this stage, but is covered by the event recalculation stage; the intermediate quantities of each batch and the aggregation completion status of the batch are persisted in the same statistical storage transaction, and retries skip confirmed batches. If they cannot be committed together, the complete data reconstruction result of the affected scope of the batch is read and then overwritten; when the same batch contains both new data and subsequent modifications to the record, the entire batch of data is applied first, and then the complete recalculation is carried out according to the affected scope of the batch to avoid using the modified amount as the new amount again; the statistical results are saved according to the business object, indicator type and statistical scope identifier to form the initial statistical results.

[0082] S6-2: Read unprocessed change events;

[0083] Read unprocessed events from the change event storage area and check whether the metric calculation field, business date field, and deletion status affect the corresponding metric.

[0084] When business dates change, the affected time windows are determined by both the old and new dates, and duplicates are removed. When the date remains unchanged but the amount changes, the window is determined using the business date corresponding to the event. When adding only new dates, the new date is used; when deleting only old dates, the old date is used. The date saved by the event is read first. Only when the record still exists and belongs to the same confirmed batch data status is the primary key used for backtracking. If the date cannot be determined, the event is retained and an error is reported, without silent skipping. The time range and affected window of each indicator under the same business object are compared item by item to determine the related indicators. This allows the original statistical range affected by deleted records or historical date corrections to be located.

[0085] Both the time window and the statistical time range of the indicators adopt an interval that includes the start point but does not include the end point; when the start point of the time window is less than the end point of the time range, and the end point of the time window is greater than the start point of the time range, the two are considered to intersect.

[0086] S6-3: Generate and execute the recalculation task;

[0087] The associated indicators are deduplicated according to business object, indicator type, and dependent time range to generate a recalculation task; window-type indicators read all valid records within the window, quarterly or cumulative indicators read all valid records within the complete time range they depend on, and internal rate of return reads the complete cash flow sequence within its evaluation range; on the confirmed data state, the corresponding adapter is executed, monetary indicators are resumed, proportional indicators are recalculated in terms of numerator and denominator, and the rate of return is recalculated to obtain the recalculation result; when there are no valid records, the monetary summation result is zero, and the proportional or rate of return returns return to an uncalculation state according to the corresponding denominator or cash flow boundary, and outdated original values ​​cannot be retained.

[0088] S6-4: Replace the initial statistical results with the recalculated results;

[0089] The original statistical results are located by identifying the business object, indicator type, and dependent time range. The corresponding result records are replaced with the recalculated results and their validity status. Unaffected results retain their original values. The replacement results form the final statistical results and are persisted for writing back in step S7.

[0090] S6-5: Retain the status of events awaiting confirmation;

[0091] The statistics module saves the association between the final statistical results and the corresponding events, and the events remain in an unprocessed state. After the idempotent write-back is completed in step S7, the status of the corresponding event is updated to processed or an acknowledgment is submitted to the message queue. If the write-back fails, the write-back of the persistent result is retried. If the write-back is successful but the acknowledgment fails, the same result is allowed to be overwritten again. Since the acknowledgment is later than the result is committed, the failure will not cause the unwritten result to lose the retry entry.

[0092] For example, if the date of a certain investment record for the same fund is changed from February to January of the same year, but the amount remains unchanged, the execution module retains the event due to the change in business dates, and the statistics module identifies January and February as the affected windows, rereads the records of these two months after synchronization, and calculates the monthly amount. If the related indicator depends on the first quarter, the complete data of the first quarter is read and the indicator is recalculated. The ratio or rate of return of the two months cannot be added together. Because the revised record is read and recalculated, the original February result removes the impact of this investment, and the January result includes the impact. The total investment amount in the first quarter remains unchanged when only the date within the quarter is changed.

[0093] For example, if a contribution amount for the same fund in the same month is revised from 1 million yuan to 1.5 million yuan, with the date remaining unchanged, the execution module saves the before and after values ​​of the amount and the corresponding business date of the event. The statistics module then locates the month based on this and recalculates the total amount based on the complete record. If other records remain unchanged and the contribution is not deleted, the total monthly paid-in capital increases by 500,000 yuan. If the event is submitted again, the same recalculated value will be overwritten, and the 500,000 yuan increase will not occur. If the record is subsequently deleted, the original window will be located based on the date before deletion, and the record will be excluded before recalculation. If there are no other valid records in the window, the amount result will be overwritten to zero. This example is a rule-based calculation and does not represent actual performance testing.

[0094] The internal rate of return (IRR) calculation module determines the discount rate that makes the net present value (NPV) of cash flows within the assessment scope zero. In this embodiment, cash flows are discrete over equal-length statistical periods, with the first period designated as period zero. The discount rate is a dimensionless ratio for each statistical period, and both net cash flows and NPV use the same currency and monetary unit. During calculation, one is first added to the current discount rate to obtain the discount base. Then, this base is raised to the power of the period number to which the cash flow belongs. The current period's net cash flow is divided by this power to obtain the current period's discount value. Finally, the discount values ​​from period zero to the last period are added together to obtain the NPV. The discount rate must be greater than -1 to prevent the discount base from being zero or from having no real meaning when extending to non-integer periods.

[0095] The calculation module reads the cash flow date, amount, and direction for the same business object, currency, and evaluation scope. Capital investment is treated as negative and recovery as positive. Records on the same day or in the same period are summed by signed amount. If the date, amount, or direction of any record is missing, an input error is returned. It does not default to zero. If the cash flow is empty or all zeros, an internal rate of return cannot be determined is returned.

[0096] Under the equal-period caliber of this embodiment, a unified period length and start date are configured. Each cash flow is assigned to its respective period according to the business date and approximated at the point in time of that period. Periods with no cash flow are filled with zeros. When the net investment in the first period is negative, and there is at least one period with positive net cash flow and the remaining net cash flows are non-negative, the net present value strictly decreases with the discount rate, and there exists a unique root-time root. This caliber gives the rate of return for each period, but it is not directly referred to as the annual rate of return. The annual rate of return is only given directly when the period length is one year. Different actual date intervals cannot be directly mixed as integer periods. If a precise annualized caliber for each day is required, the day count and annualization rules must be clearly defined separately.

[0097] For the aforementioned cash flow sequence with a negative initial value and subsequent non-negative values, an executable root-finding process is as follows: Set a lower bound greater than -1 and an upper bound greater than the lower bound. Calculate the net present value (NPV) at both ends according to the aforementioned discounting steps. When the NPV at the endpoints is zero, a candidate root is directly obtained. When the two ends have opposite signs, take the average of the upper and lower bounds as the current discount rate, calculate the midpoint NPV, retain the half-interval where the endpoint NPVs have opposite signs, and repeat the midpoint calculation. When the two ends have no opposite signs, expand the upper bound by a preset maximum number of expansions and recalculate. If the root still cannot be found, return that the root was not located within the search range, and do not use the initial guess as the result. Each calculation checks for overflow and non-finite results. If an anomaly occurs, stop and record the reason for failure. This process uses interval halving to define each update and does not rely on arbitrary addition or subtraction of the discount rate.

[0098] For example, the lower bound of the search can be set to -0.999 and the upper bound to 1. If the root is not captured, the upper bound is multiplied by two and then added by one, with a maximum expansion of 20 times. The absolute value of net present value is divided by the sum of the absolute values ​​of all net cash flows to obtain the dimensionless relative residual. 0.0000000001 is used as the relative residual tolerance, and 0.00000001 is used as the discount rate interval width tolerance, with a maximum of 256 bisections. If the midpoint net present value is zero, it can be directly successful. Otherwise, convergence is confirmed only if the relative residual and the interval width simultaneously meet the tolerance. If the maximum number of times is reached, it returns as non-convergence. A small change in the discount rate alone cannot prove that the net present value is close to zero. For non-standard sequences such as negative cash flows after the initial investment, multiple roots may appear. In this embodiment, the state that needs to be verified for the rate of return is returned, and a root is not arbitrarily selected as the only result. The above tolerance and number of times are loaded by the adapter configuration and remain unchanged within the same statistical task.

[0099] For example, if the net investment in period zero is 1 million yuan and the net recovery in period one is 1.1 million yuan, and the discount rate for each period is 10%, the discount base is 1.1. Dividing the recovery amount in period one by 1.1 gives 1 million yuan, which, when added to the negative 1 million yuan in period zero, equals zero. Therefore, the internal rate of return for this sequence is 10% per period. If the recovery is changed to 900,000 yuan, the same steps yield a zero net present value when the discount rate is negative 10% per period. If both periods are inflows, there is no valid root that makes the sum of positive cash flows after finite discount equal to zero, and the calculation module returns to an incalculable state. The above example tests the discount relationship and abnormal branches, but does not constitute a profit forecast.

[0100] The fund size calculation module in this application adopts the paid-in capital caliber, filters the capital contribution ledgers according to fund identifier and statistical scope, excludes deleted records, and sums the paid-in capital amounts; this value is the paid-in capital size and is not used in conjunction with the subscribed capital size. If the record set is empty, it returns zero; the project investment cost calculation module filters the investment ledgers according to project identifier and statistical scope, sums the corresponding investment cost amounts, and deleted records are not included in the summary.

[0101] The Limited Partner Capital Contribution Progress Calculation Module summarizes the paid-in capital of the same limited partner under the same fund, reads the subscribed capital of the partner at the same statistical point in time, divides the paid-in capital by the subscribed capital to obtain the dimensionless progress ratio, and multiplies it by 100 and adds a percentage sign when it is displayed as a percentage. When the subscribed capital is zero, negative, or missing, it returns an uncalculation status. When the excess paid-in capital results in a result greater than 100%, the actual ratio is retained and marked as abnormal, and it is not directly truncated.

[0102] The capital contribution rate calculation adapter divides the cumulative distributed income of the same investor within the statistical scope by the actual paid-in capital as of the statistical time point to obtain a dimensionless ratio; the distributed income amount adopts the source ledger income field and is not automatically included in the principal return, and the amount uses the same currency; when the actual paid-in capital amount is zero, negative or missing, it returns an uncalculation status.

[0103] The total return multiple indicator calculation adapter adds the cumulative distributed returns of the same investor to the net asset value at the same statistical point in time, and then divides it by the paid-in capital amount up to that point in time. All amounts must be consistent in terms of currency, entity, and scope. The net asset value is the value at that point in time and is not accumulated across dates. The distributed returns amount is read from the returns field of the source ledger. If the paid-in capital amount is zero, negative, or missing, it returns an uncalculation status.

[0104] The exit revenue indicator calculation adapter reads the exit amount and the corresponding investment cost based on the exit transaction identifier, and then subtracts the corresponding investment cost from the exit amount for each transaction before summing them up. For partial exits, only the cost corresponding to this exit is used, and the total cost of the project cannot be subtracted each time. If the cost correspondence is missing, the system returns to a pending verification status. If the source ledger has provided the corresponding cost field, it is read directly, and the exit corresponding cost provided by that field is used in the calculation.

[0105] S7: Write the final statistical results back to the business system database in idempotent order of the primary key.

[0106] During write-back, the target business system, target database instance, and target data table are determined based on the write-back target table information in the business object metamodel. The corresponding writer template is loaded from the template library according to the target database type. The statistical result primary key is composed of the business object, indicator type, and statistical range identifier. If a record does not exist, it is inserted; if it exists, it is updated with the complete result and status. The statistics module only confirms the corresponding event after all related results are successfully written back; otherwise, it retains a "pending retry" status. Results for the same business object are written back in batches sequentially. New batch results are only published after retries of older batches are completed to avoid older results overwriting newer results. The final statistical results and write-back target data table location information are populated into the writer template, generating a result write-back script, which is then executed. The write-back script uses idempotent write based on the primary key.

[0107] Example 2

[0108] This invention also provides a configurable data synchronization system for the private equity field, including: a storage module, a receiving module, an acquisition module, a generation module, an execution module, a statistics module, and a data access layer.

[0109] The storage module is used to store the business object metamodel; the receiving module is used to receive the synchronization task configuration.

[0110] The acquisition module is used to obtain the corresponding business object metamodel based on the business object identifier;

[0111] The generation module is used to generate read scripts and write scripts based on the business object metamodel and synchronization mode, and resolves the source logical table name through the data access layer before generating the read script;

[0112] The execution module is used to execute read and write scripts, extract, clean, transform and load data, and record log positions, read incremental logs and write to the change event storage area during incremental synchronization based on database logs.

[0113] The statistics module is used to call the indicator calculation adapter to generate and execute statistical tasks based on the statistical indicators in the business object metamodel, obtain the initial statistical results, determine the affected time window based on the unprocessed change events in the change event storage area and perform recalculation to obtain the final statistical results, and update the change event processing status.

[0114] The write-back module is used to write the final statistical results back to the business system database in an idempotent manner according to the primary key.

[0115] Example 3

[0116] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a configurable data synchronization method in the field of private equity as described in Embodiment 1.

[0117] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0118] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0119] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0120] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for synchronizing configuration data in the private equity field, characterized in that, include: The business object metamodel is constructed and stored in the storage module. The business object metamodel includes business object identifier, source logical table name, target table information, field mapping relationship, incremental synchronization strategy and statistical indicators. The statistical indicators include business date field, indicator calculation field and time window rules. Receive the synchronization task configuration, which includes the business object identifier and the synchronization mode, which is either full synchronization or incremental synchronization. Retrieve the corresponding business object metamodel based on the business object identifier; Generate read and write scripts based on the business object meta-model and synchronization mode; The scripts for reading and writing are executed to extract business data from the source database, clean and transform it according to the field mapping relationship, and then write it to the big data cluster in an idempotent manner according to the primary key. When the synchronization mode is incremental synchronization and an incremental synchronization strategy based on database logs is adopted, the log position consistent with the snapshot is recorded before the first full snapshot. After the snapshot, the incremental log is read from the position and events involving changes in the indicator calculation field or business date field are written to the change event storage area. The statistical indicators are called to the indicator calculation adapter, and the initial statistical results are obtained by incremental aggregation according to the time window. Unprocessed events are read from the change event storage area, and the affected time windows and related indicators are determined according to the unprocessed events and time window rules. The original indicator values ​​are recalculated and replaced according to the time range on which the related indicators depend, so as to obtain the final statistical results. The final statistical results are written back to the business system database in idempotent order based on the primary key.

2. The method for configurable data synchronization in the private equity field according to claim 1, characterized in that, The business object metamodel also includes write-back target table information and incremental identifier fields; the incremental synchronization strategy is either based on extraction of the incremental identifier field or incremental synchronization based on database logs; the statistical indicators also include indicator types and indicator calculation adapter identifiers; the indicator calculation fields are numerical fields that affect the indicator calculation results, and the business date field independently participates in event filtering and time window positioning; the business objects include at least one of the following: fund, limited partner, project, fundraising ledger, capital contribution ledger, investment ledger, and exit ledger; the target table information includes the target business system identifier, the target database instance identifier, and the target data table identifier; the write-back target table information includes the write-back target business system identifier, the write-back target database instance identifier, and the write-back target data table identifier; the field mapping relationship includes the correspondence between the source logical table field names and the target data table field names.

3. The method for configurable data synchronization in the private equity field according to claim 1, characterized in that, Before generating read and write scripts based on the business object metamodel and synchronization mode, the process also includes: resolving the source logical table name to the actual physical table name through the data access layer; the data access layer resolving the source logical table name to the actual physical table name includes: If the source business system has tables, the source logical table name will be resolved into multiple actual physical table names according to the table partitioning rules. If there are field renaming issues in the source business system, the source logical table field names in the business object metamodel will be resolved to the actual physical table field names according to the mapping rules.

4. The method for configurable data synchronization in the private equity field according to claim 1, characterized in that, The time window rules include the time window division units, the list of statistical time ranges for indicators, and the recalculation rules; The recalculation rules are as follows: When the business date field changes, the time windows corresponding to both the old and new values ​​are determined as the affected time windows; when the business date field remains unchanged but the indicator calculation field changes, the affected time window is determined based on the business date corresponding to the event; indicators whose calculation dependency time range intersects with the affected time window are determined as related indicators, and recalculated according to the complete time range that each related indicator depends on. Both the time window and the statistical time range of the indicators adopt an interval that includes the start point but does not include the end point; when the start point of the time window is less than the end point of the time range, and the end point of the time window is greater than the start point of the time range, the two are considered to intersect.

5. A method for configurable data synchronization in the private equity field according to claim 2, characterized in that, When the synchronization mode is incremental synchronization and the incremental synchronization strategy is extraction based on the incremental identifier field, filter conditions based on the incremental identifier field are generated in the reading script. The incremental identifier field can be a timestamp field or an auto-incrementing field; Furthermore, the following steps are included before executing the reading script: The timestamp field is used for records whose order is consistent with the order in which they are submitted and visible. The upper bound of the previous successful synchronization range is used as the lower bound of the current time, and the upper bound of the range determined at the start of the task is used as the upper bound of the current time. The auto-increment field is used to append only records that are visible in the commit order and are consistent with the auto-increment order. Records that are greater than the maximum value of the previous successful synchronization and do not exceed the upper limit determined in this operation are extracted. When the corresponding order conditions are not met or when it is necessary to capture historical updates or deletions, incremental synchronization based on database logs is used.

6. The method for configurable data synchronization in the private equity field according to claim 1, characterized in that, When the synchronization mode is incremental synchronization and the incremental synchronization strategy is incremental synchronization based on database logs, a reading script for executing a full snapshot and a log parsing configuration are generated during the initial initialization. During the recovery task, the checkpoint log position corresponding to the successfully loaded data is used. The log parsing configuration includes: log source information, subscribed database tables, event filtering rules, and field parsing mapping.

7. A method for configurable data synchronization in the private equity field according to claim 1, characterized in that, The indicator calculation adapter is pre-set with at least one calculation module for fund size, limited partner investment progress, project investment cost, exit return, invested capital dividend rate, total return multiple, and internal rate of return; the indicator calculation adapter includes an internal rate of return calculation module, which calculates the internal rate of return through a discounted cash flow iterative algorithm.

8. A method for configurable data synchronization in the private equity field according to claim 1, characterized in that, Change the event storage area to a relational database table, message queue, or structured log file in a distributed file system; The change event record includes the source table and record primary key, a list of changed fields, the old and new values ​​of the changed fields, the business date corresponding to the event, the task batch number, and the processing status. The initial processing status is unprocessed. The task batch number is generated when the synchronization task starts and is used during retries. Checkpoints where data loading and event persistence are both successful are recorded. During retries, the part confirmed at the checkpoint is skipped, and the part that has not yet been confirmed is re-executed. After the corresponding final statistical results are persisted and successfully written back, the processing status of the corresponding change event is updated to processed.

9. A configuration-based data synchronization system for the private equity field, the system being used to execute a configuration-based data synchronization method for the private equity field as described in any one of claims 1-8, characterized in that, The system includes: The storage module is used to store the meta-model of business objects; The receiving module is used to receive the synchronization task configuration; The acquisition module is used to obtain the corresponding business object metamodel based on the business object identifier; The generation module is used to generate read scripts and write scripts based on the business object metamodel and synchronization mode, and resolves the source logical table name through the data access layer before generating the read script; The execution module is used to execute read and write scripts, extract, clean, transform and load data, and record log positions, read incremental logs and write to the change event storage area during incremental synchronization based on database logs. The statistics module is used to call the indicator calculation adapter to obtain initial statistical results, determine the affected time window and related indicators based on unprocessed events, recalculate and replace the original indicator values ​​according to the time range that the related indicators depend on, and obtain the final statistical results. The write-back module is used to write back the final statistical results idempotently by the primary key; the statistics module updates the corresponding event handling status after the final statistical results are persisted and the write-back is successful.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements a configurable data synchronization method in the field of private equity as described in any one of claims 1 to 8.