Dynamic simulation method based on multi-source heterogeneous data and computer device

Abstract

The application provides a dynamic simulation method based on multi-source heterogeneous data and a computer device. The dynamic simulation method based on multi-source heterogeneous data comprises the following steps: obtaining multi-source heterogeneous input data, performing standardization processing, and adding a metadata header for describing data attributes; calling a pre-configured mapping matrix to map the multi-source heterogeneous input data to a target dimension of a unified classification caliber to obtain mapping result data; performing joint conservation verification on the mapping result data, and generating data passing the verification as a calculable state package, and performing hierarchical rollback processing on data failing the verification according to a verification failure level; performing block recursive solving using the calculable state package to generate a current round residual vector; performing adaptive convergence control based on the residual vector, and outputting the block recursive solving result after the convergence requirement is reached, so that the dynamic simulation is completed. The scheme of the application can eliminate the format difference between different data sources and ensure the stability of the solving process.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a dynamic simulation method and computer equipment based on multi-source heterogeneous data. Background Technology

[0002] With the development of big data technology, multi-source heterogeneous data processing has become a core component of complex system simulation. For example, in economic policy simulation, it is necessary to integrate heterogeneous data from multiple sources such as statistics departments, customs, energy bureaus, and environmental protection departments; in meteorological forecasting simulation, it is necessary to integrate heterogeneous data from multiple sources such as meteorological stations, satellite remote sensing, radar observation, and radiosonde stations; and in power forecasting simulation, it is necessary to integrate heterogeneous data from multiple sources such as power grids, multiple power generation departments, meteorology, and electricity load. These data differ significantly in format, scope, units, and time granularity, posing a significant challenge to data processing.

[0003] In existing technologies, dynamic simulation of multi-source heterogeneous data mainly suffers from the following problems: data from different sources requires manual field matching, unit conversion, and time alignment, which is inefficient and prone to errors, leading to inconsistent data definitions; the data verification mechanism is imperfect and cannot effectively detect structural errors and temporal anomalies in the data, resulting in unreliable data quality; the recursive solution process generally adopts iterative solution with a fixed step size, which is prone to iteration stagnation, solution oscillation, or numerical anomalies, leading to solution failure; the recursive solution uses a fully serial approach to run multiple scenarios, resulting in excessively long computation time.

[0004] Therefore, existing technologies lack dynamic simulation methods based on multi-source heterogeneous data that can solve any of the above-mentioned defects, in order to address the problems of accuracy, solution stability, and multi-scenario computation in multi-source heterogeneous data. Summary of the Invention

[0005] One object of the present invention is to provide a dynamic simulation method and computer device based on multi-source heterogeneous data that can solve at least some of the above-mentioned problems.

[0006] A further objective of this invention is to ensure the consistency of multi-source heterogeneous data, thereby providing an accurate data foundation for subsequent calculations.

[0007] Another further objective of this invention is to address the problem that existing dynamic simulation methods struggle to balance data processing quality, solution stability, and multi-scenario computational efficiency.

[0008] Specifically, this invention provides a dynamic simulation method based on multi-source heterogeneous data, comprising: Acquire multi-source heterogeneous input data, standardize the multi-source heterogeneous input data, and add metadata headers to describe data attributes; The pre-configured mapping matrix is ​​invoked to map the standardized multi-source heterogeneous input data to the target dimension with a unified classification caliber, thereby obtaining the mapping result data; Perform joint conservation verification on the mapping result data, generate computable state packets for data that pass the verification, and perform hierarchical rollback processing on data that fail the verification according to the level of verification failure. The block recursive solution is performed using a computable state package. The recursive solution is broken down into multiple modules. Variables between multiple modules are passed in a preset fixed order, and the residual information formed in each round of calculation is fed back to generate the residual vector of the current round. Adaptive convergence control is performed based on the residual vector, and the convergence requirement is determined to be met when the maximum residual of the residual vector in the current round is lower than the preset convergence threshold. Output the block recursive solution results to complete the dynamic simulation.

[0009] Optionally, during the block recursive solution process using the computable state package, the scenario calculations of multiple modules are assigned to multiple independent computing tasks for parallel processing, and each computing task is recursively processed according to the time period sequence. Multiple independent computing tasks share basic data through caching. The basic data includes: a mapping matrix, a set of core parameters for the initial state of the recursive processing, and a set of general fixed factor data.

[0010] Optionally, multiple independent computing tasks will isolate and cache the data generated during the computing process. The isolated cache will generate a cache key based on the scenario computing identifier and the running batch identifier. The scenario computing identifier is used to distinguish scenario computing, and the running batch identifier is used for the running batch of the same scenario computing. Furthermore, the number of multiple independent computing tasks will be set according to the number of CPU logical cores and dynamically adjusted according to the proportion of memory occupied by the computing.

[0011] Optionally, the cached values ​​of the isolated cache include mapping result data, the results of the joint conservation check, static parameters of the recursive processing, and the warm-start status; and the process of reading the cache includes: Use the cache key to perform cache hit verification on the cached status information; If the cache hit check passes, the cached value can be reused directly; If a cache hit check fails, further determine whether the cache is invalid. If the cache is determined to be invalid, a full recalculation is performed, and the result of the full recalculation is used to overwrite the cache.

[0012] Optionally, the steps for standardizing multi-source heterogeneous input data include: Based on preset standardization rules, heterogeneous field names, data types, and encoding systems from different data sources are uniformly converted into a set standard format; Convert similar data to the same unit and anchor them to the same time period; The data converted to a set standard format is organized into multiple functional layers according to the data function positioning. The functional layers include: base data layer, extended data layer, rule parameter layer, and dynamic recursion layer.

[0013] Optionally, the steps of performing a joint conservation check on the mapped data include: Calculate the deviation between the sum of each item in the mapping result data and the preset target total amount, and perform a total amount conservation check on the mapping result data based on the deviation; Calculate the relative row and column differences of the mapping result data, and perform a balance check on the mapping result data based on the relative row and column differences; Perform structural consistency checks on the mapping weights and boundary validity in the mapping result data; Perform time continuity verification on the variable data in the mapping result data; For all data that passes the total amount conservation check, balance check, structural consistency check, and time continuity check, write a check pass flag to generate a computable state packet.

[0014] Optionally, the steps for performing hierarchical rollback processing on data that failed verification according to the level of verification failure include: Obtain the over-limit criteria for total quantity conservation check, balance check, structural consistency check, and time continuity check, as well as the corresponding method for each over-limit criterion. The over-limit criteria include: slight over-threshold, moderate over-threshold, severe anomaly, and irreparable criteria. The rollback processing methods include: proportional recalibration, switching to a backup data source, data interpolation repair, or blocking the use of the data source.

[0015] Optionally, when the maximum residual of the residual vector in the current round is not lower than a preset convergence threshold, this method further includes: To determine whether there is iterative stagnation, iterative oscillation, or numerical anomaly, the criteria for determining iterative stagnation is that the residual decrease rate in the first set of consecutive rounds is continuously lower than the preset minimum improvement threshold. The criteria for determining iterative oscillation is that the residual component of the preset type undergoes multiple changes in direction within the second set of consecutive rounds. The criteria for determining numerical anomaly include: the value exceeds the corresponding reasonable value range, or any of the following situations occur: division by zero, non-numeric value, or infinite value. Adjust the iteration step size in cases of iteration stall or iteration oscillation; If numerical anomalies are confirmed, the abnormal data will be restricted within a reasonable range.

[0016] Optionally, this method also includes: If it is determined that the numerical anomalies or non-convergence states are concentrated in a certain module, and the number of anomalies in that module exceeds the preset threshold, the module is restarted and the recursive solution is performed again. If the convergence condition is not met after the module is restarted, a rollback is triggered, and the state of the previous time period is called back for recalculation; this continues until the convergence condition is met or the rollback limit is exceeded.

[0017] According to another aspect of the present invention, a computer device is also provided, which includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described dynamic simulation method based on multi-source heterogeneous data.

[0018] The dynamic simulation method based on multi-source heterogeneous data of this invention unifies multi-source heterogeneous data through standardization processing, eliminating format differences between different data sources; it achieves unified conversion of data with different calibers through mapping matrices, establishing a consistent calculation benchmark and providing an accurate data foundation for subsequent calculations; it ensures the quality of input data from the source through joint conservation verification and hierarchical backoff processing; the recursive solution meets the stability and accuracy requirements of recursive solution by splitting the solution process into multiple modules and having multiple modules perform variable transfer and residual summarization in a preset fixed order; and it improves the stability of the solution process through adaptive convergence control based on residual vectors.

[0019] Furthermore, the dynamic simulation method based on multi-source heterogeneous data of the present invention adopts a scheduling mechanism of parallel outer scenario and serial inner time period. Under the premise of strictly ensuring that the causal dependency relationship before and after the time period is not destroyed, it maximizes the use of multi-core computing resources. By sharing read-only basic data by multiple independent computing tasks, it avoids the repeated loading of the same basic data in different tasks and reduces the system memory occupation. At the same time, it ensures that all parallel tasks use the exact same basic data, eliminating the interference of basic data differences on the multi-scenario comparison results.

[0020] Furthermore, the dynamic simulation method based on multi-source heterogeneous data of the present invention realizes a multi-scenario parallel scheduling and cache reuse mechanism, avoiding a large amount of repetitive preprocessing and initialization calculations; through cache verification, the accuracy of reused cache data is guaranteed; through clear cache invalidation judgment conditions and old cache overwriting mechanism, invalid data is cleared in time, blocking the propagation path of erroneous cache.

[0021] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0022] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the caching process in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the standardization process of multi-source heterogeneous input data in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the joint conservation check of the mapping result data in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the convergence control process in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the computational architecture of a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention; Figure 7 This is a flowchart illustrating the process of performing multi-department heterogeneous data simulation based on a dynamic simulation method using multi-source heterogeneous data according to an embodiment of the present invention. Figure 8 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 10 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0024] Figure 1 This is a flowchart illustrating a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention; the dynamic simulation method based on multi-source heterogeneous data generally includes: Step S101: Acquire multi-source heterogeneous input data, standardize the multi-source heterogeneous input data, and add metadata headers to describe data attributes. Multi-source heterogeneous input data can include, but is not limited to, multi-source data used in economic policy simulation, meteorological forecasting simulation, and power forecasting simulation. For example, in economic policy simulation, multi-source heterogeneous input data can come from multiple sources such as statistics departments, customs, energy bureaus, and environmental protection departments, and may specifically include social accounting matrices, input-output tables, energy balance sheets, carbon emission data, pollutant emission data, trade exposure data, and policy parameters. Similarly, in meteorological forecasting simulation, multi-source heterogeneous input data can come from multiple sources such as ground meteorological stations, satellite remote sensing centers, weather radar stations, upper-air radiosonde stations, marine buoy observation networks, and meteorological reanalysis datasets, and may specifically include ground meteorological element observation data, upper-air temperature, humidity, and pressure profile data, satellite atmospheric inversion products, radar echo and precipitation estimation data, marine meteorological element observation data, gridded reanalysis data, and model physical process parameters. For example, in the field of power forecasting simulation, multi-source heterogeneous input data can come from multiple sources such as power grid dispatching departments, power generation companies, new energy power plants, meteorological service agencies, power trading centers, and power terminal acquisition systems. Specifically, it can include unit operating output data, grid node load data, wind and solar power generation data, meteorological driving factor data, user electricity load data, power trading price parameters, and grid topology parameters.

[0025] Metadata headers can include dataset identifiers, source names, version numbers, time spans, unit tags, industry coding system identifiers, and file hash values. These metadata headers are bound to the corresponding data and are present throughout the entire process of data standardization, caliber mapping, conservation verification, recursive solving, and cache reuse. They serve as the fundamental carrier for achieving data identifiability, verifiability, traceability, and reusability. Standardized multi-source heterogeneous input data can be organized into multiple functional layers according to data function positioning. These functional layers include: a baseline data layer, an extended data layer, a rule parameter layer, and a dynamic recursive layer.

[0026] Step S102: Call the pre-configured mapping matrix to map the standardized multi-source heterogeneous input data to the target dimension with a unified classification caliber, and obtain the mapping result data.

[0027] The mapping matrix is ​​a pre-configured core tool for achieving unified transformation of multi-source heterogeneous data. It is a two-dimensional numerical matrix composed of source classification, target classification, and mapping weights. As globally shared read-only basic data, all simulation scenarios use the same set of mapping rules, fundamentally ensuring the consistency of the benchmark for multi-scenario comparative analysis. Taking economic policy simulation as an example, the industry mapping matrix and product-to-industry mapping matrix can be called to map data from various source classifications to the target dimension of standardized industry classifications (e.g., 153 departments nationwide).

[0028] Step S103: Perform a joint conservation check on the mapping result data, generate a computable state packet for the data that passes the check, and perform a layered rollback process for the data that fails the check according to the failure level. The joint conservation check includes multiple checks, such as: total quantity conservation check, balance check, structural consistency check, and time continuity check.

[0029] Step S104 involves using a computable state package to perform a block-based recursive solution. The recursive solution is divided into multiple modules, with variables passed between modules in a preset fixed order. The residual information generated in each round of calculation is fed back to generate the residual vector for the current round. The scheduling rule for the recursive solution is scenario-parallel processing followed by time-period serial processing. Within each module, the computational scenario task is processed recursively year by year according to the time-period sequence, ensuring causal dependencies in the time dimension.

[0030] Step S105: Perform adaptive convergence control based on the residual vector, and determine that the current round has reached convergence if the maximum residual of the residual vector in the current round is lower than the preset convergence threshold. Adaptive convergence control may include: convergence master criterion control, stagnation criterion control, oscillation criterion control, etc., to ensure the stability of the recursive solution.

[0031] Step S106: Output the block recursive solution results to complete the dynamic simulation. The block recursive solution results can include simulation results for each year. The output results can also include: verification logs, solution logs, and traceability reports, so that the output results can correspond one-to-one with specific scenarios, specific time periods, and specific solution logs, thereby supporting anomaly location, process review, and replay recalculation.

[0032] The method in this embodiment unifies multi-source heterogeneous data through standardization processing, eliminating format differences between different data sources; it achieves unified conversion of data with different calibers through a mapping matrix, establishing a consistent computational benchmark and providing an accurate data foundation for subsequent calculations; it ensures the quality of input data from the source through joint conservation verification and hierarchical backoff processing; it breaks down the recursive solution into multiple modules, with variables of each module being passed unidirectionally in a preset fixed order, ensuring the accuracy and stability of the solution; and it improves the stability of the solution process through adaptive convergence control based on residual vectors.

[0033] In this embodiment, during the block recursive solution process using computable state packages, the scenario calculations of multiple modules are allocated to multiple independent computing tasks for parallel processing. Within each computing task, recursive processing is performed according to time period order. Multiple independent computing tasks share basic data through caching. The time period can be an annual period, and within each year, computing tasks perform scenario calculations in parallel. Subsequently, each scenario is processed serially according to year order. The scheduling topology for block recursive solution can employ a two-level scheduling. The first level is scenario-level parallelism, allocating different policy scenarios to independent worker threads or processes; the second level is time period (e.g., by year) serial processing, where each scenario is strictly recursively processed according to time period order, prohibiting cross-time period parallelism to ensure causal propagation of variables. Each scenario executes in a fixed block order within each time period.

[0034] The basic data includes: a mapping matrix, a set of core parameters for recursively processing the initial state (also known as base period parameters), and a set of general fixed factor data (also known as static parameter blocks). Scenario calculation tasks share this read-only basic data.

[0035] Multiple independent computation tasks isolate and cache the data generated during the computation process. The isolated cache generates a cache key based on the scenario computation identifier (scenario_id) and the run batch identifier (run_id). In other words, write-type states (such as the solution results, logs, and cache namespaces for a single time period) are isolated by scenario_id + run_id to avoid concurrent write pollution.

[0036] The number of independent computing tasks (parallelism limit) is set based on the number of CPU logical cores and dynamically adjusted according to the proportion of memory consumed by the computation. In some embodiments, the parallelism limit max_parallel_scenarios can be set to min(number of CPU logical cores - 1, preset limit P), that is, the parallelism limit is the smaller value between the number of CPU logical cores minus one and the preset limit P. The preset limit P can be set according to the configuration of the computing system and the computing resources consumed.

[0037] When the available system memory falls below a preset memory threshold (memory_low_watermark), the number of parallel tasks is dynamically reduced. The memory threshold (memory_low_watermark) can be set to 20% (the specific value can be adjusted according to the specific computing scenario).

[0038] The isolated cache contains mapping result data, joint conservation check results, static parameters of recursive processing, and warm-start status. Mapping result data is a set of intermediate data generated from standardized, heterogeneous input data from multiple sources, after a unified conversion using a pre-configured mapping matrix. Joint conservation check results are the complete results generated after performing total conservation checks, balance checks, structural consistency checks, and time continuity checks on the mapping result data; they serve as the basis for determining whether the data is qualified for simulation. Static parameters of recursive processing refer to the set of core parameters that remain fixed throughout the entire recursive dynamic simulation cycle and do not change with time or scenario variations. Warm-start status is a snapshot of the recursive dynamic solution process; it refers to the complete set of data automatically saved by the system after convergence at a certain time, containing all necessary intermediate states, parameters, and context information. As a core component of the isolated cache, it is used to achieve automatic anomaly recovery.

[0039] In an embodiment of economic policy simulation, the static parameters for recursive processing may include: basic economic system parameters (capital depreciation rate, investment adjustment coefficient, propensity to consume, etc.), technical parameters (input-output coefficient, energy conversion efficiency, emission factor, etc.), physical constants and general conversion coefficients (standard coal conversion coefficient, currency conversion coefficient, etc.), and basic system operation parameters (convergence threshold, maximum number of iterations, verification threshold, etc.). The hot-start state may include: the convergence and solution status of each functional module, all intermediate variable values ​​passed between modules, solver status information, metadata and verification information, convergence logs, and exception handling records.

[0040] Figure 2 This is a flowchart illustrating cache processing in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. The process of reading the cache may include: Step S201: Use the cache key to perform a cache hit check on the cached status information. In some embodiments, the cache key may include: data version, mapping version, parameter version, scenario identifier, and time period.

[0041] Step S202: If the cache hit check passes, the cached value is directly reused. Hot start reuse requires the same scenario, adjacent years, and the same constraint configuration. Only when the scenario_id is consistent, the previous period's results pass the time-loop review, and there are no structural changes to the scenario and constraint configuration, will the key endogenous variable vector from the previous period be used as the initial value for this year. If any condition is not met, the process reverts to cold start.

[0042] Step S203: If the cache hit verification fails, further determine whether the cache is invalid. The cache is determined to be invalid when there is a change in input version, a change in rule version, a verification failure, or when it is recorded in the cache invalidation log.

[0043] In step S204, if the cache is determined to be invalid, a full recalculation is performed, and the result of the full recalculation is used to overwrite the cache, thereby preventing the erroneous cache from continuing to spread.

[0044] The above-mentioned cache reading process significantly improves the running efficiency and reliability of recursive dynamic simulation while ensuring data consistency and calculation accuracy through multi-dimensional verification and strict reuse mechanism. In multi-scenario batch operation, the mapping matrix, base period calibration results, static parameter blocks and the previous year's hot start status are cached and reused. The outer layer is parallel according to scenario and the inner layer is serial according to time period scheduling method to improve computing throughput.

[0045] Figure 3 This is a flowchart illustrating the standardization process of multi-source heterogeneous input data in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. The steps for standardizing multi-source heterogeneous input data may include: Step S301 involves converting heterogeneous field names, data types, and encoding systems from different data sources into a unified standard format based on preset standardization rules. This step eliminates differences in surface formats between different data sources by uniformly converting all heterogeneous data into the system standard format based on a preset global standardization rule base. The standardized field attributes and encoding information are then written to the metadata header.

[0046] Step S302 involves converting similar data to the same unit and anchoring them to the same time period. This step eliminates differences in data dimensions and time granularity, ensuring comparability of similar data from different sources by using the same unit. Anchoring data to the same time period aggregates data from different time granularities, such as monthly and quarterly, into a unified annual granularity, and uses the base year as an anchor point to appropriately fill in missing time points, ensuring the continuity of the time series.

[0047] Step S303 involves organizing the data, converted to a set standard format, into multiple functional layers based on data function positioning. These functional layers include: a baseline data layer, an extended data layer, a rule parameter layer, and a dynamic recursive layer. The baseline data layer stores globally shared core basic data; the extended data layer stores extended dimension data such as environment and energy; the rule parameter layer stores rule data such as mapping and verification; and the dynamic recursive layer stores parameters for the evolution of the control system. Each layer achieves a unidirectional dependency relationship where the lower layers are independent and the upper layers depend on each other, supporting incremental data updates, layered verification, and multi-granularity cache reuse, providing support for data sharing and isolation during parallel operation in multiple scenarios.

[0048] In the embodiment of economic policy simulation, the baseline data layer, extended data layer, rule parameter layer, and dynamic recursion layer can be respectively positioned as: Layer A (benchmark accounting layer), Layer B (environment extension layer), Layer C (policy mapping layer), and Layer D (dynamic recursion layer).

[0049] Figure 4 This is a flowchart illustrating the joint conservation check of mapping result data in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. The steps for performing the joint conservation check of the mapping result data may include: Step S401: Calculate the deviation between the sum of each item in the mapping result data and the preset target total amount, and perform a total amount conservation check on the mapping result data based on the deviation.

[0050] Step S402: Calculate the relative difference between rows and columns of the mapping result data, and perform a balance check on the mapping result data based on the relative difference between rows and columns.

[0051] Step S403: Perform a structural consistency check on the mapping weights and boundary validity in the mapping result data.

[0052] Step S404: Perform time continuity verification on the variable data in the mapping result data.

[0053] Step S405: Write a verification pass flag to all data that has passed the total amount conservation check, balance check, structural consistency check, and time continuity check, so as to generate a computable state packet.

[0054] The above-mentioned total quantity conservation check calculates the relative deviation between the sum of each component data in the mapping result data and the target total quantity, and determines whether the total quantity conservation requirement is met based on the deviation. The balance check calculates the relative differences between rows and columns of matrix data in the mapping result data, such as the relative difference between the row sum and column sum of each account in the social accounting matrix, to determine whether the balance requirement is met. The structural consistency check checks whether the mapping weights in the mapping result data are complete and whether the sum of all mapping weights meets the requirements; it also checks whether parameter values ​​are within legal boundary ranges, such as whether the share parameter is between 0 and 1. The time continuity check checks whether there are abnormal growth rate jumps or intensity spikes in the time series variables in the mapping result data, determining whether the time series is continuous and smooth.

[0055] Only when all four checks mentioned above pass is a check pass flag written to the data, and a computable state packet is generated. The computable state packet encapsulates all the state vectors, parameter blocks, version information, and check summaries required for input, serving as the sole input for block-based recursive solution.

[0056] The process of hierarchically rolling back data that fails verification according to the failure level may include: obtaining the over-limit criteria for total quantity conservation verification, balance verification, structural consistency verification, and time continuity verification, as well as the corresponding rollback processing method for each over-limit criterion. The over-limit criteria include: slight over-threshold, moderate over-threshold, severe anomaly, and irreparable criteria. The rollback processing methods include: proportional recalibration, switching to a backup data source, data interpolation repair, or blocking the use of the data source. Those skilled in the art can set the numerical proportions exceeding the threshold for slight over-threshold, moderate over-threshold, and severe anomaly over-threshold according to the needs of dynamic simulation calculations.

[0057] For example, for data that slightly exceeds the threshold, data interpolation or proportional recalibration (simultaneously scaling up or down the values ​​of each department according to a uniform ratio to realign the total) can be used for repair; for data that moderately exceeds the threshold, the system can automatically switch to a backup data source and re-execute the preprocessing and mapping process; for severely abnormal data, structural correction or time-series smoothing repair can be performed; for unrepairable data, the system can block the use of the data source, terminate the current data processing flow, and output an anomaly log.

[0058] The aforementioned proportional recalibration can simultaneously scale up or down the values ​​of each department by a uniform ratio while preserving the original relative structure and share proportions of the data, thus realigning the totals. Switching to a backup data source can stop using the current abnormal primary data source and call the backup data source pre-configured by the system. Data interpolation and repair can use compliant interpolation, fitting, smoothing, and imputation algorithms to locally correct the data and repair abnormal segments. Blocking the data source is the final handling method for unrepairable anomalies such as completely invalid data, multiple serious failures in multiple validations, and failure to meet standards after multiple repairs. It forcibly intercepts the current problematic data source, prohibits it from entering the subsequent block recursive solution process, and records the anomaly log.

[0059] The aforementioned joint conservation check performs a comprehensive verification of the mapping result data. Only data that passes all four checks can generate a computable state package to enter the solution process, thus avoiding calculation errors caused by poor data from the source.

[0060] Figure 5 This is a flowchart illustrating the convergence control process in a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention. The steps for adaptive convergence control during the recursive solution process include: Step S501: Perform the convergence criterion judgment, that is, enter the convergence check state, and determine whether the maximum residual of the residual vector in the current round is lower than the preset convergence threshold. If yes, then execute step S512 to determine to enter the convergence acceptance state. If no, then sequentially determine whether there is iteration stagnation, iteration oscillation or numerical anomaly.

[0061] Step S502: Perform the determination of iteration stagnation, that is, determine that the residual decrease rate of the first set rounds is continuously lower than the preset minimum improvement threshold. If it is determined that the iteration is stagnant, proceed to step S505.

[0062] Step S503: Perform the iterative oscillation judgment, that is, determine whether the residual component of the preset type has changed direction multiple times within the second consecutive set round. If it is determined to be iterative oscillation, proceed to step S505.

[0063] Step S504: Perform a numerical anomaly judgment. The criteria for judging numerical anomalies include: the value exceeds the corresponding reasonable value range, or any of the following situations occur: division by zero, non-numeric value, or infinite value. The reasonable value range can be configured according to the specific meaning of the value and the normal data range. If the numerical anomaly is determined, proceed to step S506.

[0064] In step S505, if there is iteration stagnation or iteration oscillation, adjust the iteration step size and perform a damped update, i.e., enter the damped adjustment state. If the residual vector decreases continuously and stably, the iteration step size can also be appropriately increased to improve iteration efficiency.

[0065] Step S506: If a numerical anomaly is determined, the abnormal data is restricted within a reasonable numerical range, and the source module of the abnormal data is identified. Numerical anomalies can include invalid logarithms, division by zero, non-numeric values, or infinite values. For example, when a numerical anomaly is detected, variable boundary correction is performed to correct the abnormal data to a reasonable boundary value, i.e., entering a boundary truncation state.

[0066] Step S507: Determine whether there are numerical anomalies or non-convergence states concentrated in a certain module, and whether the number of anomalies in that module exceeds a preset threshold. If so, proceed to step S508 to restart the module.

[0067] Step S508: Restart the abnormal modules that have exceeded the limit of abnormal data or have not converged, and recursively solve the problem again, that is, enter the block restart state, and determine whether the convergence condition has been met after the module restarts. Step S509: If the convergence condition is not met after the module restarts, a rollback is triggered, and the state of the previous time period is called back for recalculation, i.e., the rollback state is entered until the convergence condition is met or the rollback count is exceeded.

[0068] Step S510: Determine if the number of rollbacks has exceeded the limit. If so, execute the hard termination state in step S511. Step S511: Enter hard termination state. In this state, the system writes a hard termination log and terminates the current scenario calculation task, thus completing the blocking.

[0069] Step S512: Enter the convergence acceptance state. In this state, the system writes the convergence pass log and enters the annual result writing process.

[0070] In some embodiments, the preset convergence threshold can be set within a relatively strict range, and the minimum improvement threshold can be set to a level that facilitates the identification of stagnation. The first set number of rounds (number of consecutive stagnation rounds), the second set number of rounds (length of the oscillation observation window), the number of direction changes (number of oscillation triggers), the upper and lower bounds of the damping coefficient, the damping decrease amplitude, the damping increase amplitude, the block restart threshold, and the upper limit of the number of backoffs can all be set according to the data scale and model complexity. The above parameters can be adjusted according to the data scale, but the control logic and triggering sequence remain unchanged.

[0071] The above process constructs a closed-loop control system consisting of primary criteria, secondary criteria, and hierarchical control actions. The control actions are triggered in the following order: damping update, boundary truncation, block restart, annual rollback, and blocking. The damping update can employ a method of reducing the step size when convergence is poor and appropriately increasing the step size when convergence is stable.

[0072] In the aforementioned recursive dynamic processing, the recorded fields may include: batch identifier, scenario identifier, time period, iteration round, maximum residual of the current round, stall count, oscillation count, processing action code, values ​​before and after damping coefficient adjustment, number of block restarts, number of rollbacks, recovery flag, and time consumption. The dynamic simulation method based on multi-source heterogeneous data in this embodiment can also provide a result traceability and verification report generation mechanism, enabling each output indicator to be associated with source data, mapping rules, parameter versions, verification logs, and solution logs, thereby improving the auditability, verifiability, and accountability of the results.

[0073] In a recursive dynamic simulation computing system that covers multiple departments and policy scenarios across the country, the dynamic simulation method based on multi-source heterogeneous data in this embodiment can meet the engineering application requirements of multi-department recursive dynamic simulation computing scenarios in terms of unified input of multi-source heterogeneous data, stable recursive dynamic solution, efficient operation in multiple scenarios, and traceable output of results. Figure 6 This is a schematic diagram of the computational architecture of a dynamic simulation method based on multi-source heterogeneous data according to an embodiment of the present invention.

[0074] The computing architecture can be summarized as a three-layer, six-module structure, including a data consistency layer L1, a recursive solution layer L2, and a result traceability layer L3. The data consistency layer L1 comprises: a data access and standardization module M1, a caliber mapping and unit unification module M2, and a conservation verification and rollback module M3. The recursive solution layer L2 comprises: a block-based recursive solution module M4, and a convergence control and parallel caching module M5. The result traceability layer L3 comprises a result traceability and reporting module M6.

[0075] The data access and standardization module M1 is used to acquire multi-source heterogeneous input data, including a 153-sector social accounting matrix, CO2 emission data, pollutant emission data, governance data, trade exposure data, CBAM (Carbon Border Adjustment Mechanism) coverage list, ETS (Emissions Trading System) coverage and quota data, and dynamic parameter data such as capital, investment, and labor. This module performs field name standardization, encoding standardization, null value handling, type correction, and metadata header generation for each data source. The input data is organized hierarchically according to the baseline accounting layer (layer A), environmental extension layer (layer B), policy mapping layer (layer C), and dynamic recursion layer (layer D).

[0076] The module M2, which maps data by caliber and unifies units, calls a preset mapping matrix to uniformly map source-caliber industry data to the target dimension of 153 departments. For product-level HS6 data, it generates department-level export exposure share to Europe through a product-to-department mapping matrix and performs unified conversion for different units such as amount, emissions, and taxes. In addition, this module performs time anchoring and missing value imputation on data from different years, using the set year as the base period anchor point for consistency processing.

[0077] The conservation verification and rollback module M3 is used to perform joint verification on the mapping results and standardization results. The verification includes total amount conservation verification, account balance verification, structural consistency verification, and time continuity verification. When the verification passes, a computable state package is generated; when the verification fails, it performs processing according to four levels: slight over-threshold, moderate over-threshold, severe anomaly, and irreparable, such as proportional recalibration, switching to backup source, interpolation repair, or blocking input to the model.

[0078] The block-based recursive solution module M4 solves the simulation model using an annual recursive approach. It treats the core states that need updating each year as a unified annual state set, which includes key variables such as domestic commodity prices, output, capital returns, and external price scaling factors. The annual solution process is divided into four sub-blocks: Price and Absorption Block, Production and Factor Block, Emissions and Policy Cost Block, and Trade Allocation and Market Clearing Block. Variables are passed between blocks in a predetermined order: the Price and Absorption Block outputs price information and passes it to the Production and Factor Block; the Production and Factor Block outputs intermediate inputs, added value, and energy inputs and passes them to the Emissions and Policy Cost Block; the Emissions and Policy Cost Block outputs policy unit costs and passes them to the Trade Allocation and Market Clearing Block; the Trade Allocation and Market Clearing Block outputs market residual information and feeds this residual information back to the updater for the next iteration. In some implementations, policy unit costs can also influence price updates in the Price and Absorption Block.

[0079] The block-based recursive solution module M4 can execute according to the following process: The Price and Absorption Block takes into account the price and exchange rate information from the previous round, calculates the updated price variables and absorption-side constraints, and outputs the price results and absorption-side intermediate quantities for the next block; the Production and Factors Block takes into account the price results and absorption-side intermediate quantities output from the previous block, calculates output, labor demand, capital demand, factor returns, and energy input, and outputs intermediate results for subsequent blocks; the Emissions and Policy Costs Block takes into account the output results, energy input, policy coverage, and price parameters, calculates carbon emissions, pollutant emissions, and policy unit costs, and outputs emission results and policy cost results; the Trade Allocation and Market Clearing Block takes into account the price results, output results, and policy cost results, calculates trade allocation, exports to Europe, and market residuals, and outputs residual information for convergence determination; the residual information formed in the current round is submitted to the convergence control and parallel caching module M5 for convergence determination; if the convergence condition has not yet been met, the next iteration is entered based on the updated control parameters. When the convergence condition is met for a given year, the results for that year are written, and a hot-start state for the next year is generated.

[0080] The convergence control and parallel caching module M5 performs repeatable convergence determination, parameter updates, rollback recovery, and cache scheduling. This module transforms uncontrollable failures into a detectable, adjustable, and recoverable controlled process through adaptive convergence control; by sharing a cache and performing parallel scheduling, it maintains cross-year state consistency and improves throughput. This enhances the stability, repeatability, and engineering usability of recursive dynamic simulation calculations under complex constraints.

[0081] The Result Traceability and Reporting module M6 outputs iterative solution results, verification logs, rollback logs, solution logs, and traceability information. Specific information may include annual macroeconomic, industry, trade, emission, and policy cost indicators. It also outputs conservation error, balance error, share error, iteration count, residual sequence, failure retry records, and traceability information such as data source version, mapping version, and parameter version. The output results can be matched one-to-one with specific scenarios, specific years, and specific solution logs, thus supporting anomaly location, process review, and replay recalculation.

[0082] Figure 7 This is a flowchart illustrating a dynamic simulation method for multi-department heterogeneous data based on multi-source heterogeneous data according to an embodiment of the present invention. The process of this multi-department heterogeneous data simulation may include: Step S701: Obtain multi-source heterogeneous input data. The input data can be various heterogeneous basic data required for economic policy simulation, such as social accounting matrix (SAM), carbon emission data, pollutant emission data, governance data, trade and policy mapping data, dynamic path parameters, etc., covering multiple sources such as statistics, customs, energy, and environment.

[0083] Step S702 involves performing field standardization, unit conversion, and time alignment. This step performs basic standardization processing on multi-source heterogeneous data to unify heterogeneous field names, data types, and encoding systems; it also converts similar data into system standard units to uniformly anchor data with different time granularities to the annual granularity, thereby eliminating data table heterogeneity.

[0084] Step S703: Perform industry mapping and product-to-industry mapping, call the pre-configured mapping matrix, and uniformly convert data with different statistical calibers into the system standard 153 department target calibers, including industry caliber mapping and product-to-industry mapping, and perform preliminary mapping verification.

[0085] Step S704: Perform joint verification. Perform four-level joint verification on the mapping result data. If all verifications pass, proceed to step S705; if any verification fails, proceed to step S710. The four-level joint verification includes: total amount conservation verification, used to verify the consistency between the sum of the sub-items and the authoritative macro total amount; account balance verification, used to verify the row and column balance relationship of matrix data such as SAM matrix; structural consistency verification, used to verify the integrity of the mapping weights and the legality of the parameter boundaries; and time continuity verification, used to verify the rationality of the changing trend of time series data.

[0086] Step S705: Generate a computable state package. Package all the data that has passed the verification into a unified computable state package to build a consistent computational state. This package contains all the input information required for model solving and serves as the only valid input for subsequent recursive solutions.

[0087] Step S706: Perform block recursive solution by year. A block solution architecture is adopted, breaking down the entire solution process into multiple independent functional modules, which are solved recursively year by year. Parallel computing by department and region is supported, and variables are passed unidirectionally between modules in a fixed causal order.

[0088] Step S707: Perform convergence control and determine if convergence has occurred. Perform multi-dimensional convergence checks on the solution results of the current round, including: main convergence check (whether the maximum residual is below a threshold), convergence stall check, convergence oscillation check, and numerical anomaly check, comprehensively diagnosing various abnormal states in the solution process. If convergence fails, proceed to step S720. If convergence occurs, proceed to step S708, entering the next stage of multi-scenario parallelism and cache reuse.

[0089] Step S708, the next stage of multi-scenario parallelism and cache reuse. This step adopts a scheduling mechanism of parallel computation of the outer scenario and serial execution of the inner year-progressive process, running multiple policy simulation scenarios simultaneously; by reusing mapping results, base period calibration results, static parameter blocks, and hot start states through an isolated cache mechanism, redundant calculations are significantly reduced, and the efficiency of multi-scenario operation is improved.

[0090] Step S709: Output results, logs, and traceability information. Output the final simulation results, including macro and industry results, emission and policy cost results, etc.; at the same time, output end-to-end traceability information, including verification logs, rollback logs, solution logs, and traceability chain information, to meet the auditability requirements of policy evaluation.

[0091] Step S710: Perform data hierarchical rollback repair. For minor exceedances, perform proportional recalibration; for moderate exceedances, switch to a backup data source; for severe anomalies, perform interpolation repair; for unrepairable verification errors, block input. After repair, return to the corresponding preceding processing steps and re-execute.

[0092] Step S720: Convergence Improvement and Module Restart. Perform corresponding actions based on the type of non-convergence. For example, for iteration stagnation or oscillation, perform damped updates; for numerical anomalies, perform data boundary truncation; for single-module anomalies, perform block restarts; for severe anomalies, perform annual rollbacks. After performing the improvement actions, re-execute the block recursive solution.

[0093] The technical effectiveness of the method in this embodiment is verified using the following metrics: convergence success rate (the percentage of tasks that meet the convergence conditions in a given set of scenarios), failure recovery rate (the percentage of tasks that resume convergence after triggering a rollback), average iteration rounds (the average number of iterations required to reach convergence in each year), redundant computation compression rate (the percentage of preprocessing and initialization computations that are skipped after cache reuse), and traceability integrity rate (the proportion of output metrics that can be linked back to the data version, mapping version, parameter version, verification log, and rollback log). These metrics demonstrate the comprehensive technical improvement of this solution in terms of stability, efficiency, and auditability.

[0094] The above steps are introduced using economic policy simulation as an example. Those skilled in the art can apply the method of this embodiment to fields such as meteorological forecasting simulation, power forecasting simulation, and recursive processing of multi-source heterogeneous data as needed. By standardizing the processing, multi-source heterogeneous data can be unified, eliminating format differences between different data sources. By adaptive convergence control based on residual vectors, the stability of the solution process can be improved.

[0095] This embodiment also provides a computer program product 110, a computer-readable storage medium 120, and a computer device 114. Figure 8This is a schematic diagram of a computer program product 110 according to an embodiment of the present invention. Figure 9 This is a schematic diagram of a computer-readable storage medium 120 according to an embodiment of the present invention. Figure 10 This is a schematic block diagram of a computer device 114 according to an embodiment of the present invention.

[0096] Computer program product 110 includes computer program 111, which, when executed by processor 310, implements the steps of the dynamic simulation method based on multi-source heterogeneous data described above. Computer-readable storage medium 120 stores the computer program 111 thereon, which, when executed by processor 310, implements the steps of the dynamic simulation method based on multi-source heterogeneous data described above. Computer device 114 may include memory 320, processor 310, and computer program 111 stored in memory 320 and running on processor 310.

[0097] The computer program 111 used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages.

[0098] Computer program 111 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of the invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of computer-readable program instructions.

[0099] For the purposes of this embodiment, computer program product 110 is a related product that includes computer program 111.

[0100] For the purposes of this embodiment, computer-readable storage medium 120 is a tangible device capable of holding and storing computer program 111. It can be any device capable of containing, storing, communicating, propagating, or transmitting program 111 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 120 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0101] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A dynamic simulation method based on multi-source heterogeneous data, characterized in that... include: Acquire multi-source heterogeneous input data, perform standardization processing on the multi-source heterogeneous input data, and add metadata headers to describe data attributes; The pre-configured mapping matrix is ​​invoked to map the standardized multi-source heterogeneous input data to a target dimension with a unified classification caliber, thereby obtaining the mapping result data; Perform a joint conservation check on the mapping result data, generate a computable state packet for the data that passes the check, and perform hierarchical rollback processing on the data that fails the check according to the check failure level. The computable state package is used to perform block recursive solution, which is divided into multiple modules. The variables between the multiple modules are passed in a preset fixed order, and the residual information formed in each round of calculation is fed back to generate the residual vector of the current round. Adaptive convergence control is performed based on the residual vector, and when the maximum residual of the residual vector in the current round is lower than the preset convergence threshold, it is determined that the convergence requirement has been met in this round. Output the block-based recursive solution results to complete the dynamic simulation. The step of performing joint conservation verification on the mapping result data includes: calculating the deviation between the sum of each item in the mapping result data and the preset target total amount, and performing total amount conservation verification on the mapping result data based on the deviation; calculating the relative row and column differences of the mapping result data, and performing balance verification on the mapping result data based on the relative row and column differences; performing structural consistency verification on the mapping weights and boundary legality in the mapping result data; performing time continuity verification on the variable data in the mapping result data; and writing a verification pass flag for all data that passes the total amount conservation verification, the balance verification, the structural consistency verification, and the time continuity verification, so as to generate the computable state packet. The step of performing layered rollback processing on the data that failed the verification according to the verification failure level includes: obtaining the over-limit standards of the total amount conservation verification, the balance verification, the structural consistency verification, and the time continuity verification, as well as the rollback processing method corresponding to each over-limit standard. The over-limit standards include: slight over-limit, moderate over-limit, severe abnormality, and irreparable criteria. The rollback processing methods include: proportional recalibration, switching to a backup data source, data interpolation repair, or blocking the use of the data source.

2. The dynamic simulation method based on multi-source heterogeneous data according to claim 1, characterized in that, During the block recursive solution process using the computable state package, the scenario calculations of the multiple modules are assigned to multiple independent computing tasks for parallel processing, and the recursive processing is performed in the order of time periods within each computing task. The multiple independent computing tasks share basic data through caching. The basic data includes: the mapping matrix, the core parameter set of the initial state of the recursive processing, and the general fixed factor data set.

3. The dynamic simulation method based on multi-source heterogeneous data according to claim 2, characterized in that, The multiple independent computing tasks isolate and cache the data generated during the computing process. Each isolated cache generates a cache key based on a scenario computing identifier and a run batch identifier. The scenario computing identifier distinguishes different scenarios, and the run batch identifier is used for the same run batch of the scenario computing. The number of the multiple independent computing tasks is set according to the number of CPU logical cores and dynamically adjusted according to the proportion of memory occupied by the computing.

4. The dynamic simulation method based on multi-source heterogeneous data according to claim 3, characterized in that, The cached value of the isolated cache includes the mapping result data, the result of the joint conservation check, the static parameters of the recursive processing, and the warm start status; and the process of reading the cache includes: Use the cache key to perform cache hit verification on the cached status information; If the cache hit check passes, the cached value can be reused directly; If the cache hit check fails, further determine whether the cache is invalid; If the cache is determined to be invalid, a full recalculation is performed, and the result of the full recalculation is used to overwrite the cache.

5. The dynamic simulation method based on multi-source heterogeneous data according to claim 1, characterized in that, The step of standardizing the multi-source heterogeneous input data includes: Based on preset standardization rules, heterogeneous field names, data types, and encoding systems from different data sources are uniformly converted into a set standard format; Convert similar data to the same unit and anchor them to the same time period; The data converted to a set standard format is organized into multiple functional layers according to the data function positioning. The functional layers include: base data layer, extended data layer, rule parameter layer, and dynamic recursion layer.

6. The dynamic simulation method based on multi-source heterogeneous data according to claim 1, characterized in that, When the maximum residual of the residual vector in the current round is not lower than a preset convergence threshold, the method further includes: The system determines whether there is iteration stagnation, iteration oscillation, or numerical anomaly. The criteria for determining iteration stagnation are that the residual decrease rate is continuously lower than the preset minimum improvement threshold for the first set of consecutive rounds. The criteria for determining iteration oscillation are that the residual component of the preset type undergoes multiple changes in direction within the second set of consecutive rounds. The criteria for determining numerical anomaly include: the value exceeds the corresponding reasonable value range, or any of the following situations occur: division by zero, non-numeric value, or infinite value. If the iteration stalls or oscillations occur, adjust the iteration step size; If the numerical anomaly is confirmed, the abnormal data is restricted within the reasonable numerical range.

7. The dynamic simulation method based on multi-source heterogeneous data according to claim 6, characterized in that, Also includes: If it is determined that the numerical anomalies or non-convergence states are concentrated in a certain module, and the number of anomalies in that module exceeds a preset threshold, the module is restarted to perform recursive solution again. If the convergence condition is not met after the module is restarted, a rollback is triggered, and the state of the previous time period is called back for recalculation; this continues until the convergence condition is met or the rollback limit is exceeded.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the dynamic simulation method based on multi-source heterogeneous data as described in any one of claims 1 to 7.

Images