Method for reconciling JSON data and data warehouse base table data

Abstract

The application provides a reconciliation method for JSON data and data warehouse base table data, which comprises the following steps: receiving reconciliation scheduling information for controlling task execution, reconciliation mapping relationship for associating JSON data and data warehouse base table data, and alarm configuration; constructing reconciliation execution instructions for data comparison according to a preset template and the reconciliation mapping relationship; triggering distributed computing resources to load JSON data and data warehouse base table data and performing data comparison based on the reconciliation scheduling information and the reconciliation execution instructions, and returning reconciliation result data; and analyzing the reconciliation result data according to the alarm configuration to generate corresponding alarm results. Thus, the application realizes data consistency checking between heterogeneous data and fixed table data without relying on manual customization, eliminates the structural bottleneck formed by the fact that the existing method is difficult to efficiently and accurately complete comparison in a large-scale reconciliation scenario, and improves reconciliation efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for reconciling JSON data and data warehouse baseline data. Background Technology

[0002] With the rapid development of big data and cloud computing, enterprises and institutions are increasingly relying on data with flexible structures such as JSON for business processing and analysis. JSON data typically supports complex structures such as nested objects, arrays, and optional fields, while data warehouses usually employ a two-dimensional table storage model with fixed column names and data types. This difference in data organization makes ensuring consistency between JSON data and data warehouse data during business processes, reporting, and data analysis a critical issue that needs to be addressed.

[0003] Existing methods typically involve manually writing SQL statements or custom scripts to expand JSON data and compare it field-by-field with the data warehouse table. This approach not only requires customized development for different JSON structures and data warehouse tables, resulting in high adaptation costs, but also often takes too long when dealing with massive amounts of data at the terabyte level daily, making it difficult to meet real-time or near real-time data reconciliation requirements.

[0004] Therefore, existing methods have failed to establish a unified abstraction and efficient processing mechanism between heterogeneous data structures and fixed table structures, resulting in data comparison relying on manual customization and being difficult to complete quickly and accurately in large-scale scenarios, thus forming a structural bottleneck in data consistency verification. Summary of the Invention

[0005] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the technical deficiency of the structural bottleneck in data consistency verification in the prior art.

[0006] Firstly, this application provides a method for reconciling JSON data and data warehouse baseline data, the method including:

[0007] Receive reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse baseline data, and alarm configurations;

[0008] Based on the preset template and reconciliation mapping relationship, construct reconciliation execution instructions for data comparison;

[0009] Based on the reconciliation scheduling information and reconciliation execution instructions, the distributed computing resources are triggered to load JSON data and data warehouse bottom table data, and data comparison is performed to return the reconciliation result data;

[0010] Based on the alarm configuration, the reconciliation result data is analyzed to generate corresponding alarm results.

[0011] In one embodiment, the step of constructing a reconciliation execution instruction for data comparison based on a preset template and reconciliation mapping relationship includes:

[0012] Load preset template;

[0013] Parse the reconciliation mapping relationship to obtain the reconciliation mapping parameters, and assign the reconciliation mapping parameters to the entity class. The reconciliation mapping parameters include JSON data identifier, data warehouse base table data identifier, data filtering conditions, JSON field extraction rules, data warehouse base table field concatenation rules, and offset time information.

[0014] Replace the corresponding placeholders in the preset template with the variables in the entity class to obtain the reconciliation execution instruction used for data comparison.

[0015] In one embodiment, the step of parsing the reconciliation mapping relationship to obtain the reconciliation mapping parameters includes:

[0016] Read the configuration fields in the reconciliation mapping relationship;

[0017] Based on the field identifier of each configuration field and its position in the configuration structure of the reconciliation mapping relationship, the parameter type of each configuration field is identified.

[0018] Based on the parameter type identification results, each configuration field is structured and parsed to determine the JSON data identifier, data warehouse base table data identifier, data filtering conditions, JSON field extraction rules, data warehouse base table field concatenation rules, and offset time information, so as to obtain the reconciliation mapping parameters.

[0019] In one embodiment, based on reconciliation scheduling information and reconciliation execution instructions, the steps of triggering distributed computing resources to load JSON data and data warehouse baseline data, and performing data comparison include:

[0020] Analyze the reconciliation scheduling information to determine the task scheduling parameters used to control the execution of reconciliation tasks;

[0021] The reconciliation execution instructions and task scheduling parameters are encapsulated to generate a distributed computing task description;

[0022] The task description is sent to the distributed computing resource through the task submission interface, which triggers the distributed computing resource to execute the reconciliation execution instruction according to the task scheduling parameters, so as to load JSON data and data warehouse bottom table data and perform data comparison.

[0023] In one embodiment, the step of analyzing reconciliation result data and generating corresponding alarm results based on alarm configuration includes:

[0024] Retrieve the alarm trigger threshold from the alarm configuration;

[0025] Analyze the number of reconciliation discrepancies in the reconciliation results data;

[0026] The number of reconciliation discrepancies is compared with the alarm trigger threshold, and an alarm result corresponding to the comparison result is generated.

[0027] In one embodiment, the method further includes:

[0028] Based on the reconciliation results, generate trend charts and detailed reports;

[0029] When a result query request is received, a trend chart and a detailed table are displayed.

[0030] Secondly, this application provides a reconciliation device for JSON data and data warehouse baseline data, the device comprising:

[0031] The reconciliation configuration receiving module is used to receive reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse bottom table data, and alarm configurations.

[0032] The reconciliation execution instruction construction module is used to construct reconciliation execution instructions for data comparison based on preset templates and reconciliation mapping relationships;

[0033] The reconciliation result data acquisition module is used to trigger distributed computing resources to load JSON data and data warehouse bottom table data based on reconciliation scheduling information and reconciliation execution instructions, perform data comparison, and return reconciliation result data;

[0034] The alarm result generation module is used to analyze the reconciliation result data based on the alarm configuration and generate corresponding alarm results.

[0035] In one embodiment, the reconciliation execution instruction construction module includes:

[0036] The preset template recording unit is used to load preset templates;

[0037] The reconciliation mapping parameter determination unit is used to parse the reconciliation mapping relationship, obtain the reconciliation mapping parameters, and assign the reconciliation mapping parameters to the entity class. The reconciliation mapping parameters include JSON data identifier, data warehouse bottom table data identifier, data filtering conditions, JSON field extraction rules, data warehouse bottom table field concatenation rules, and offset time information.

[0038] The reconciliation execution instruction determination unit is used to replace the corresponding placeholders in the preset template with variables in the entity class to obtain the reconciliation execution instruction for data comparison.

[0039] Thirdly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of any of the reconciliation methods for JSON data and data warehouse bottom table data in the above embodiments.

[0040] Fourthly, this application provides a computer device, including: one or more processors, and a memory;

[0041] The memory stores computer-readable instructions that, when executed by one or more processors, perform the steps of any of the reconciliation methods for JSON data and data warehouse bottom table data as described in the above embodiments.

[0042] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0043] The reconciliation method for JSON data and data warehouse baseline data provided in this application receives reconciliation scheduling information and reconciliation mapping relationships, and constructs a unified reconciliation execution instruction based on a preset template. This transforms JSON data and data warehouse baseline data into data processing objects following the same instruction structure at the execution level, thereby establishing a unified abstraction and processing method between heterogeneous data structures and fixed table structures. This avoids the problem of manually customizing comparison logic for different data structures. Simultaneously, based on the reconciliation scheduling information, distributed computing resources are triggered to load and execute the reconciliation execution instruction, enabling the data comparison process to be completed efficiently under a unified mechanism, meeting the needs of fast and accurate reconciliation in large-scale scenarios. Furthermore, the reconciliation result data is analyzed according to alarm configurations to generate alarm results, allowing the reconciliation results to be automatically identified and fed back. Therefore, this application achieves data consistency verification between heterogeneous data and fixed table data without relying on manual customization, eliminating the structural bottleneck caused by the difficulty of efficient and accurate comparison in large-scale reconciliation scenarios in existing methods, and improving reconciliation efficiency and accuracy. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating the reconciliation method for JSON data and data warehouse baseline data provided in this application embodiment;

[0046] Figure 2A schematic diagram of the structure of the reconciliation device for JSON data and data warehouse bottom table data provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] This application provides a method for reconciling JSON data and data warehouse baseline data. The following embodiments illustrate this method using a computer device as an example. It is understood that the computer device can be any device with data processing capabilities, including but not limited to a single server, server cluster, personal laptop, desktop computer, etc. Figure 1 As shown, the method may include the following steps:

[0050] S101: Receives reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse baseline data, and alarm configurations.

[0051] Among them, reconciliation scheduling information refers to structured control information used to describe the timing, cycle, or triggering conditions of reconciliation tasks, which is used to constrain the initiation and execution of reconciliation tasks; reconciliation mapping relationship refers to structured configuration information used to describe the correspondence between JSON data fields and data warehouse bottom table data fields, which is used to establish the data association foundation between different data structures; alarm configuration refers to rule information used to describe the rules that need to be triggered when the reconciliation results are abnormal, which is used to limit the conditions and scope of anomaly identification.

[0052] In the specific implementation process, the system first receives reconciliation scheduling information, reconciliation mapping relationships, and alarm configurations from external systems, and then performs unified parsing and processing on the received information. The reconciliation scheduling information, upon receipt, is parsed into scheduling parameters used to control the execution of reconciliation tasks, ensuring that subsequent reconciliation processing is triggered when predetermined conditions are met. The reconciliation mapping relationships, upon receipt, are parsed into field-level association descriptions, used to clarify the field correspondence between JSON data and data warehouse baseline data, thus establishing a clear association foundation for subsequent data processing. The alarm configurations, upon receipt, are parsed into identifiable rule descriptions, used to identify data anomalies requiring attention.

[0053] Reconciliation scheduling information, reconciliation mapping relationships, and alarm configurations are stored separately as control information and rule information that can be invoked in subsequent steps, and logically form a unified configuration set. By centrally receiving and processing the above information in the same stage, the reconciliation task has complete and clear control conditions, data association rules, and anomaly judgment criteria before execution, thereby ensuring that subsequent processing can be carried out in a consistent configuration environment.

[0054] By centrally receiving reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse baseline data, and alarm configurations before the reconciliation task is executed, the task execution conditions, data association rules, and anomaly judgment criteria are clearly defined and solidified before processing begins. This avoids the uncertainty caused by dynamically determining relevant information during execution. Since the reconciliation scheduling information defines the execution method and timing of the task, the reconciliation mapping relationship clarifies the data association basis between different data structures, and the alarm configuration pre-defines the abnormal situations that need to be identified, the centralized reception of these elements ensures that subsequent reconciliation processing is carried out under consistent constraints. This improves the stability and consistency of the overall processing, reduces manual intervention, and provides a fundamental guarantee for the reliable execution of reconciliation tasks and the effective output of results.

[0055] S102: Based on the preset template and reconciliation mapping relationship, construct the reconciliation execution instruction for data comparison.

[0056] Among them, the preset template refers to the structured rule information used to uniformly describe the structure of the data comparison instruction, and is used to limit the organization and semantic position of various fields in the reconciliation execution instruction; the reconciliation execution instruction refers to the structured instruction data generated based on the preset template and combined with the reconciliation mapping relationship, and is used to clarify the data fields that need to be compared in the subsequent data comparison process and their corresponding relationships.

[0057] In the specific implementation process, after obtaining the preset template and reconciliation mapping relationship, the preset template is first parsed to determine the overall structure and field organization rules of the reconciliation execution instructions used for data comparison. The preset template predefines the arrangement order and expression method of various fields in the reconciliation execution instructions, ensuring structural consistency of the instructions. Subsequently, the reconciliation mapping relationship is parsed to identify the correspondence between JSON data fields and data fields in the data warehouse baseline table, and this correspondence is filled into the preset template one by one, so that the abstract fields in the template are replaced by specific data field descriptions.

[0058] As the preset template and reconciliation mapping relationship are gradually integrated, the reconciliation execution instructions for data comparison are constructed. These instructions can simultaneously describe the data fields that need to be compared in both JSON data and data warehouse baseline data, maintaining a consistent instruction structure. In this way, the constructed reconciliation execution instructions transform the data comparison requirements under different data structures into a unified instruction description, enabling subsequent data comparison processes to be carried out directly based on these instructions without needing to define comparison rules separately for different data structures. The constructed reconciliation execution instructions are saved in a directly callable data format to support the execution of subsequent data comparison processing.

[0059] By constructing reconciliation execution instructions based on preset templates and reconciliation mapping relationships, the field correspondences and instruction structures required for data comparison are uniformly described and solidified before execution. This ensures that data comparison needs under different data structures can be expressed in a consistent instruction format, thereby avoiding repeated parsing or manual confirmation of field relationships during the data comparison process. Since the data fields involved in the comparison and their correspondences are clearly defined during the generation stage of the reconciliation execution instructions, subsequent data comparisons can be directly executed according to the instructions. This improves the consistency and controllability of the data comparison process, reduces manual configuration and processing discrepancies, and provides a fundamental guarantee for the stable and efficient execution of data comparisons.

[0060] S103: Based on the reconciliation scheduling information and reconciliation execution instructions, trigger the distributed computing resources to load JSON data and data warehouse bottom table data, perform data comparison, and return the reconciliation result data.

[0061] Distributed computing resources refer to a computing environment composed of multiple computing nodes, used for parallel processing of large-scale data; reconciliation result data refers to the structured result information generated after comparing JSON data with data warehouse bottom table data, used to reflect the consistency between different data.

[0062] In the specific implementation process, after obtaining the reconciliation scheduling information and reconciliation execution instructions, the triggering time and execution conditions of the reconciliation task are first determined based on the reconciliation scheduling information, and the corresponding data processing flow is initiated when the conditions are met. Subsequently, according to the data range and field information described in the reconciliation execution instructions, the distributed computing resources are triggered to load the corresponding JSON data and data warehouse baseline data. Since the data fields involved in the comparison and their corresponding relationships are clearly defined in the reconciliation execution instructions, the targeted reading and organization of data can be completed according to the instructions during the data loading process, preparing for subsequent comparison processing.

[0063] After data loading is complete, the loaded JSON data and data warehouse baseline data are compared item by item according to the pre-defined data comparison rules in the reconciliation execution instruction. The comparison process is executed in parallel with the support of distributed computing resources, enabling different data shards to participate in the comparison operation simultaneously, thereby improving overall processing efficiency. After the data comparison is completed, the comparison results generated by each computing node are summarized and organized to generate structured reconciliation result data, which reflects the consistency or differences between the JSON data and the data warehouse baseline data. This reconciliation result data is returned as the output of this step.

[0064] By triggering distributed computing resources to uniformly load JSON data and data warehouse baseline data based on reconciliation scheduling information and reconciliation execution instructions, and performing data comparison under predetermined instruction constraints, the data loading and comparison process is conducted under clear execution control and consistent rule guidance, thus avoiding disordered triggering or duplicate processing. Since distributed computing resources can complete data loading and comparison operations in parallel, data comparison efficiency is improved. Furthermore, the data range and comparison rules are clearly defined before execution of the reconciliation execution instructions, ensuring consistency and controllability in the comparison process. Therefore, reconciliation result data can be stably generated in large-scale data scenarios, ensuring the accuracy and repeatability of the output results and providing a reliable data foundation for subsequent processing.

[0065] S104: Analyze the reconciliation result data according to the alarm configuration and generate the corresponding alarm results.

[0066] Among them, the alarm result refers to the structured output information generated when the reconciliation result data meets the preset conditions in the alarm configuration, which is used to characterize abnormal reconciliation situations and their corresponding alarm status.

[0067] In the specific implementation process, after obtaining the reconciliation result data, the alarm configuration is parsed to extract the defined anomaly judgment conditions and rules. Subsequently, the parsed alarm configuration is correlated with the reconciliation result data, and each comparison result in the reconciliation result data is analyzed item by item to determine whether it meets the corresponding anomaly conditions in the alarm configuration. During the analysis, the reconciliation result data is matched and filtered according to the judgment methods specified in the alarm configuration, ensuring that abnormal data can be accurately identified.

[0068] When one or more data items in the reconciliation result data meet the abnormal conditions defined in the alarm configuration, a corresponding alarm result is generated to record the abnormality type and its occurrence status. The generated alarm result is output in structured data format, making it directly usable by subsequent processing flows, thus completing the transformation from reconciliation result data to alarm result. Through this method, the reconciliation result data is transformed into alarm results with clear abnormal indications, making the output of data consistency verification more identifiable.

[0069] By analyzing reconciliation results data based on alarm configurations and generating corresponding alarm results, the anomaly identification process after reconciliation is completed is automatically carried out according to pre-defined rules, thus avoiding inconsistencies caused by manual, item-by-item judgment of reconciliation results. Since the anomaly judgment conditions are clearly defined in the alarm configurations before analysis, the reconciliation results data can be processed under a unified standard, ensuring consistency and repeatability in the anomaly identification process. This improves the efficiency and accuracy of anomaly detection and allows the reconciliation results to be output in a more easily identifiable form, providing direct evidence for subsequent processing.

[0070] In the above embodiments, by receiving reconciliation scheduling information and reconciliation mapping relationships, and constructing unified reconciliation execution instructions based on preset templates, JSON data and data warehouse bottom table data are transformed into data processing objects following the same instruction structure at the execution level. This establishes a unified abstraction and processing method between heterogeneous data structures and fixed table structures, avoiding the problem of manually customizing comparison logic for different data structures. Simultaneously, based on the reconciliation scheduling information, distributed computing resources are triggered to load and execute the reconciliation execution instructions, enabling the data comparison process to be completed efficiently under a unified mechanism, meeting the needs of fast and accurate reconciliation in large-scale scenarios. Furthermore, the reconciliation result data is analyzed according to alarm configurations, and alarm results are generated, enabling the reconciliation results to be automatically identified and fed back. Therefore, this application achieves data consistency verification between heterogeneous data and fixed table data without relying on manual customization, eliminating the structural bottleneck caused by the difficulty of efficient and accurate comparison in large-scale reconciliation scenarios in existing methods, and improving reconciliation efficiency and accuracy.

[0071] In one embodiment, the step of constructing a reconciliation execution instruction for data comparison based on a preset template and reconciliation mapping relationship includes:

[0072] Load preset template;

[0073] Parse the reconciliation mapping relationship to obtain the reconciliation mapping parameters, and assign the reconciliation mapping parameters to the entity class. The reconciliation mapping parameters include JSON data identifier, data warehouse base table data identifier, data filtering conditions, JSON field extraction rules, data warehouse base table field concatenation rules, and offset time information.

[0074] Replace the corresponding placeholders in the preset template with the variables in the entity class to obtain the reconciliation execution instruction used for data comparison.

[0075] Among them, the reconciliation mapping parameters refer to the specific set of parameters parsed from the reconciliation mapping relationship, which are used to carry the key information for data comparison; the entity class refers to the structured data object used to carry the reconciliation mapping parameters, and its internal variables correspond one-to-one with the placeholders in the preset template; the JSON data identifier is used to uniquely identify the source of the JSON data participating in the reconciliation, the data warehouse table data identifier is used to uniquely identify the data warehouse table participating in the reconciliation, the data filtering conditions are used to limit the range of data participating in the reconciliation, the JSON field extraction rules are used to describe the way to obtain the target fields from the JSON data, the data warehouse table field concatenation rules are used to describe the way the fields in the data warehouse table are combined, and the offset time information is used to describe the time alignment rules involved in the reconciliation process.

[0076] In the specific implementation process, a preset template is first loaded, and its contents are read and parsed to obtain the fixed structure and variable placeholder positions used to construct the reconciliation execution instructions. The preset template predefines the instruction framework for data comparison, ensuring consistency in instruction structure across different reconciliation tasks and providing a unified foundation for subsequent parameter filling. After loading the preset template, the parsing phase of the reconciliation mapping relationship begins.

[0077] When parsing the reconciliation mapping relationship, the association information between the JSON data described in the configuration and the data in the data warehouse's underlying table is broken down into several reconciliation mapping parameters, and the parsed parameters are extracted separately. Specifically, the JSON data identifier and data warehouse underlying table data identifier used to identify the data participating in the reconciliation are identified from the reconciliation mapping relationship to limit the data source participating in the reconciliation; at the same time, data filtering conditions are extracted to limit the data range participating in the data comparison; further, the JSON field extraction rules and data warehouse underlying table field concatenation rules are parsed to clarify the acquisition and combination methods of fields under different data structures; and offset time information is obtained to align the data range of different data sources in the time dimension. After the above reconciliation mapping parameters are parsed, they are uniformly assigned to the entity class, so that the entity class completely carries all the parameter information required for a data comparison.

[0078] After the entity class is constructed, variables in the entity class are replaced one by one with the corresponding placeholders according to the defined positions in the preset template. This replacement process fills the abstract placeholder content in the preset template with specific reconciliation mapping parameters, thereby generating a complete and definite reconciliation execution instruction. The generated reconciliation execution instruction can simultaneously describe the source, filtering conditions, field extraction methods, and field combination rules of both JSON data and data warehouse baseline data, providing a direct execution basis for subsequent data loading and data comparison.

[0079] By first loading a preset template to fix the overall structure of the reconciliation execution instructions, and then parsing the reconciliation mapping relationship into reconciliation mapping parameters containing JSON data identifiers, data warehouse table data identifiers, data filtering conditions, JSON field extraction rules, data warehouse table field concatenation rules, and offset time information, and uniformly assigning these parameters to entity classes, the scattered reconciliation rules can be centralized into a clearly structured set of parameters. Then, reconciliation execution instructions for data comparison are generated through variable substitution. This processing method separates the instruction structure from the specific parameters. The reconciliation execution instructions have clear data sources, field processing rules, and time alignment information from the generation stage, thus avoiding the repeated construction of instruction logic or manual concatenation of rules. This helps ensure the consistency and reusability of the reconciliation execution instructions, reduces configuration complexity, and provides a reliable foundation for the stable and accurate execution of the data comparison process.

[0080] In one embodiment, the step of parsing the reconciliation mapping relationship to obtain the reconciliation mapping parameters includes:

[0081] Read the configuration fields in the reconciliation mapping relationship;

[0082] Based on the field identifier of each configuration field and its position in the configuration structure of the reconciliation mapping relationship, the parameter type of each configuration field is identified.

[0083] Based on the parameter type identification results, each configuration field is structured and parsed to determine the JSON data identifier, data warehouse base table data identifier, data filtering conditions, JSON field extraction rules, data warehouse base table field concatenation rules, and offset time information, so as to obtain the reconciliation mapping parameters.

[0084] Among them, configuration fields refer to data units used to express specific reconciliation rules in the reconciliation mapping relationship. Each configuration field has a corresponding field identifier and configuration position. Field identifier refers to the marking information used to distinguish the semantic types of different configuration fields. Configuration structure position refers to the hierarchical or positional characteristics of the configuration field in the overall structure of the reconciliation mapping relationship. Parameter type identification refers to the process of determining the parameter category to which the configuration field belongs based on the field identifier and configuration structure position.

[0085] In the specific implementation process, the reconciliation mapping relationship is first read and processed, and each configuration field describing the data reconciliation rules is retrieved one by one. The reconciliation mapping relationship is stored in a structured form, and different configuration fields are used to express information with different semantics. By reading sequentially or traversally, the content of all configuration fields contained in the mapping relationship can be completely obtained. At the same time, when reading each configuration field, the field identifier corresponding to that configuration field and its configuration position in the overall structure of the reconciliation mapping relationship are also obtained to provide necessary context information for subsequent processing.

[0086] After reading the configuration fields, parameter type identification is performed on each field based on its field identifier and its position in the reconciliation mapping relationship. The field identifier reflects the semantic characteristics of the configuration field, while the position in the configuration structure reflects its hierarchy and scope within the overall mapping relationship. By comprehensively judging both, the parameter type corresponding to the configuration field can be determined. This identification process logically distinguishes configuration fields with different purposes, thus avoiding ambiguity caused by relying solely on the field content for vague parsing.

[0087] After parameter type identification is completed, structured parsing is performed on each configuration field based on the identification results. For configuration fields identified as related to data sources, identification information is obtained to uniquely identify JSON data and data warehouse baseline data; for configuration fields identified as conditional, data filtering conditions for limiting participation in reconciliation are obtained; for configuration fields identified as rule-based, JSON field extraction rules and data warehouse baseline field concatenation rules are obtained; and for configuration fields identified as time-based, offset time information describing time offset relationships is obtained. Through this parsing process, the original configuration fields are converted into semantically clear and structurally sound parameter content.

[0088] After parsing, the obtained JSON data identifiers, data warehouse base table data identifiers, data filtering conditions, JSON field extraction rules, data warehouse base table field concatenation rules, and offset time information are uniformly organized into a reconciliation mapping parameter set. This parameter set can fully reflect the mapping rules required for a data reconciliation, providing direct and clear parameter inputs for the construction of subsequent reconciliation execution instructions. This ensures that subsequent processing can be carried out under clear and definite rule constraints, thereby guaranteeing the continuity and consistency of the data reconciliation process.

[0089] By first reading the configuration fields in the reconciliation mapping relationship, then combining field identifiers and configuration structure positions to identify the parameter types of the configuration fields, and performing structured parsing based on this, different semantic configurations originally mixed in the same mapping relationship can be accurately distinguished and transformed into clear reconciliation mapping parameters. This processing method ensures that JSON data identifiers, data warehouse bottom table data identifiers, data filtering conditions, JSON field extraction rules, data warehouse bottom table field concatenation rules, and offset time information have clear types and definite meanings during the generation stage. This avoids parsing ambiguity or parameter confusion caused by unclear configuration semantics, thereby improving the accuracy and consistency of the reconciliation mapping parameter generation process and providing a stable and reliable foundation for the correct construction of subsequent reconciliation execution instructions and data comparison processing.

[0090] In one embodiment, based on reconciliation scheduling information and reconciliation execution instructions, the steps of triggering distributed computing resources to load JSON data and data warehouse baseline data, and performing data comparison include:

[0091] Analyze the reconciliation scheduling information to determine the task scheduling parameters used to control the execution of reconciliation tasks;

[0092] The reconciliation execution instructions and task scheduling parameters are encapsulated to generate a distributed computing task description;

[0093] The task description is sent to the distributed computing resource through the task submission interface, which triggers the distributed computing resource to execute the reconciliation execution instruction according to the task scheduling parameters, so as to load JSON data and data warehouse bottom table data and perform data comparison.

[0094] Among them, task scheduling parameters refer to the set of parameters obtained from the reconciliation scheduling information that are used to control the execution order, triggering time or resource usage of tasks; distributed computing task description refers to the task description information formed by uniformly encapsulating the reconciliation execution instructions and task scheduling parameters, which is used to fully express the execution requirements of a distributed reconciliation task.

[0095] In the specific implementation process, the received reconciliation scheduling information is first parsed and processed. This information includes details describing when the reconciliation task is triggered, how it is executed, and its execution priority. By reading and parsing the fields of this scheduling information, key control information used to control the execution of the reconciliation task is extracted and converted into task scheduling parameters. These parameters can then be directly used to control the execution of subsequent reconciliation tasks, thus providing the foundation for the orderly operation of the reconciliation tasks.

[0096] After determining the task scheduling parameters, the generated reconciliation execution instructions and task scheduling parameters are uniformly encapsulated. The reconciliation execution instructions describe the range of JSON data and data warehouse table data to be loaded, as well as the data comparison method. The task scheduling parameters describe the execution constraints and scheduling conditions of the reconciliation task. By integrating the two into a distributed computing task description, this task description can simultaneously and completely express the execution logic and execution control requirements of the reconciliation task, thus forming a standardized task input suitable for distributed computing environments.

[0097] After generating the distributed computing task description, it is sent to the distributed computing resources via the task submission interface. Upon receiving the task description, the distributed computing resources control the execution timing and method of the reconciliation task based on the task scheduling parameters it contains, and load the corresponding JSON data and data warehouse baseline data according to the reconciliation execution instructions. Subsequently, data comparison processing is performed with the collaborative work of each computing node, enabling the reconciliation task to run stably in the distributed environment according to the predetermined scheduling rules, and finally completing the data comparison operation between the JSON data and the data warehouse baseline data.

[0098] First, the reconciliation scheduling information is parsed to determine the task scheduling parameters, ensuring that the reconciliation task has a clear execution timing, order, and constraints before execution, thus avoiding disordered triggering or resource contention during operation. Then, the reconciliation execution instructions and task scheduling parameters are encapsulated to generate a distributed computing task description, ensuring that data comparison logic and execution control requirements are uniformly expressed within the same task carrier. This guarantees that the distributed computing resources can obtain complete and consistent execution information at once when receiving the task. Finally, the task description is sent to the distributed computing resources through the task submission interface and executed according to the task scheduling parameters. This allows the distributed computing resources to load JSON data and data warehouse baseline data under controlled conditions and complete data comparison processing. This achieves schedulable execution and stable operation of the reconciliation task in a distributed environment, improving the controllability and overall efficiency of reconciliation execution.

[0099] In one embodiment, the step of analyzing reconciliation result data and generating corresponding alarm results based on alarm configuration includes:

[0100] Retrieve the alarm trigger threshold from the alarm configuration;

[0101] Analyze the number of reconciliation discrepancies in the reconciliation results data;

[0102] The number of reconciliation discrepancies is compared with the alarm trigger threshold, and an alarm result corresponding to the comparison result is generated.

[0103] Among them, the alarm trigger threshold refers to the numerical boundary used to measure the degree of abnormality in the reconciliation results, and is used to represent the maximum number of reconciliation differences that are allowed to exist; the number of reconciliation differences refers to the number of inconsistent records obtained from the reconciliation result data, and is used to characterize the degree of deviation in data comparison.

[0104] In this embodiment, after completing the reconciliation process between JSON data and data warehouse baseline data, the generated reconciliation result data can be further analyzed to support automatic alarm determination of the reconciliation results. First, the alarm trigger threshold is read from the pre-configured alarm configuration. This threshold is used to clarify the anomaly tolerance range of the current reconciliation task in the result determination stage, so that the subsequent analysis process has a unified judgment benchmark.

[0105] After obtaining the alarm trigger threshold, the generated reconciliation result data is traversed and analyzed. Statistical processing is performed on the recorded data differences to obtain the number of reconciliation differences reflecting the overall deviation level of this reconciliation. This process, through centralized analysis of the reconciliation result data, aggregates the difference information scattered across multiple comparison records into a quantifiable numerical result, facilitating subsequent unified judgment.

[0106] After determining the number of reconciliation discrepancies, this value is compared with an alarm trigger threshold. Based on the comparison result, a corresponding alarm is generated. When the number of reconciliation discrepancies reaches or exceeds the alarm trigger threshold, an alarm indicating an abnormal state is generated; when the number of reconciliation discrepancies does not reach the alarm trigger threshold, an alarm indicating a normal state is generated. This method allows the reconciliation results to be directly converted into clear alarm judgment information, ensuring consistency with specific alarm configurations.

[0107] By first obtaining the alarm trigger threshold in the alarm configuration, and then statistically analyzing the number of reconciliation discrepancies in the reconciliation result data and comparing it with the alarm trigger threshold, the judgment process of the reconciliation result has a clear and quantifiable basis for judgment. This avoids the problem that the reconciliation result is only presented in the form of raw data and it is difficult to directly judge the degree of abnormality. At the same time, the discrepancy analysis results are uniformly compared with the pre-set thresholds and corresponding alarm results are generated, so that the alarm judgment process is consistent and repeatable, and the accuracy and automation of the anomaly identification of the reconciliation result are improved.

[0108] In one embodiment, the method further includes:

[0109] Based on the reconciliation results, generate trend charts and detailed reports;

[0110] When a result query request is received, a trend chart and a detailed table are displayed.

[0111] The trend chart refers to a visual graph generated based on time-related or batch-related information in the reconciliation results data, used to reflect the changing trend of the number or changes in reconciliation discrepancies over time or the number of executions. The detail table refers to tabular data formed after the reconciliation results data is organized according to a predetermined field structure, used to display specific reconciliation records and their corresponding discrepancies. The result query request refers to a query instruction or request initiated by the user to obtain and view the reconciliation analysis results.

[0112] In this embodiment, after obtaining the reconciliation results data, the data can be further organized and processed to support subsequent result analysis and display. First, the reconciliation results data is aggregated and categorized, summarizing data reflecting the quantity or proportion of discrepancies according to time or reconciliation batches, thereby generating trend data that reflects overall changes. Based on this, the trend data is graphically transformed to form a trend chart, allowing the changes in the reconciliation results to be presented intuitively.

[0113] Meanwhile, each discrepancy record in the reconciliation results can be organized according to a predetermined data structure, extracting relevant field information and generating a detailed table in a preset order. This method preserves the complete details of the reconciliation results while ensuring a consistent data format for the displayed content, facilitating subsequent queries and analysis.

[0114] After generating the trend charts and detailed tables, the results can be stored or cached. When a result query request is received, the generated trend charts and detailed tables are directly retrieved and displayed based on the request. This eliminates the need for repeated calculations of the original reconciliation data during the result query process, ensuring the timeliness and stability of the display process.

[0115] By generating trend charts and detailed tables based on reconciliation results data, abstract reconciliation results are transformed into intuitive trend information and traceable details. This facilitates a quick grasp of overall changes and supports in-depth viewing of specific discrepancies. At the same time, when a result query request is received, the generated trend charts and detailed tables are directly displayed, avoiding redundant processing of reconciliation result data. This improves the efficiency of result querying and display, and enhances the usability and readability of reconciliation result analysis.

[0116] To facilitate understanding of the scheme in this application, specific examples are provided below.

[0117] In this example, the platform provides a method for configuring mapping rules for reconciling JSON data with data warehouse baseline data, allowing users to configure reconciliation rules through a visual interface. During configuration, users fill in basic information, mapping relationship information, and alarm configuration information, and submit the configuration after completion. The platform saves the configuration information and calls an external offline data platform interface to create a corresponding data comparison Spark task for subsequent data reconciliation processing.

[0118] The basic information describes the scheduling attributes and rule identifiers of the reconciliation rules. Users configure Cron expressions to determine the execution time of the Spark scheduled task for data comparison in the external offline data platform; the Cron expression controls the task's execution cycle. The basic information also includes user-defined rule names and descriptions to distinguish different data reconciliation rules and facilitate management. This information is primarily used for the platform's own scheduling management and is passed to the external offline data platform when the Spark task is created.

[0119] The mapping relationship information is used to define the two underlying tables involved in the reconciliation and the specific data comparison method. Users select Validation Table 1 and Validation Table 2 on the mapping relationship configuration page. The selected underlying tables must be pre-registered in the underlying table data source configuration before they can be displayed. For each of the two validation tables, users can configure data filtering conditions. The syntax of the filtering conditions is consistent with the WHERE clause in SQL. For example, the conditional expression xxx=1 AND yyy=2 can be used to limit the range of data involved in the reconciliation.

[0120] In reconciliation scenarios involving JSON data, users can select a high-dimensional JSON field as the data source for reconciliation, such as the `msg` field. Based on this, users select specific key-value pairs from this JSON field to be verified, such as `aa`, `bb`, and `cc`, to generate the data content for comparison in Verification Table 1. For non-JSON format business wide tables, users select the corresponding fields in Verification Table 2, such as `ee`, `ff`, and `gg`, and concatenate these fields into a string according to the configured order to generate the data content for comparison in Verification Table 2; for example, concatenating `ee`, `ff`, and `gg` into `ee|ff|gg`.

[0121] The mapping information also includes time offset configurations to control the actual data time used for reconciliation. When time placeholder variables are used in the SQL, users can configure the offset unit and offset amount. For example, when the offset unit is day and the offset amount is -1, a reconciliation task executed on May 18, 2017 will automatically retrieve data from May 17, 2017. The offset unit can be configured as day or hour, and is usually consistent with the data granularity of the underlying table itself. For example, choose day when the underlying table is a day-level task, and choose hour when the underlying table is an hour-level task.

[0122] The platform encapsulates the user configuration information into entity classes based on the fields filled in the mapping relationship. These entity classes maintain the variables that need to be replaced in the SQL template. The fields stored in the entity classes include Verification Table 1 fields, Verification Table 2 fields, field extraction rules, field concatenation rules, filtering conditions, and time offset information. When a scheduled task is triggered, the platform reads the configuration information of the corresponding rules from its own database, assigns values ​​to the fields in the entity classes, and replaces the values ​​selected by the user in the mapping relationship with the reserved variable positions in the Spark SQL template, thereby generating a complete Spark SQL query for data comparison.

[0123] The generated Spark SQL is used to perform data comparison between the two verification tables. When the platform calls the external offline data platform interface to create a Spark task, it passes the SQL generated in the entity class as the task execution content, and combines it with the Cron expression configured in the basic information to make the Spark task execute automatically according to the set period. During execution, Spark loads data from the two verification tables, compares field values, and writes the comparison results to the Hive table corresponding to the rule after the task is completed. The method of creating tasks using an external offline data platform is used because the amount of data involved in the reconciliation is large, requiring the distributed computing capabilities of Spark, and the data and storage of the verification tables are located on the offline data platform.

[0124] After the Spark task completes, the platform retrieves the reconciliation discrepancy data based on the task results and processes it in conjunction with the alarm configuration. The alarm configuration defines the alarm trigger threshold and alarm notification method. When the number of reconciliation discrepancies meets the configured trigger conditions, the platform generates the corresponding alarm result. Alarm notification methods can include office system notifications or instant messaging notifications, and the corresponding notification addresses can be configured. Users can also choose not to configure alarms and only retain the function of querying reconciliation results.

[0125] When a user clicks the report entry in the rule list on the platform's front-end page, the platform displays the reconciliation results view corresponding to that rule. This view shows a trend chart and a detailed data table. The trend chart uses data from the inconsistency count and execution date information saved by the platform during each task execution, reflecting the trend of reconciliation differences over time. The detailed data table displays the specific field names and values ​​where differences exist. This data comes from the Hive table written by the Spark task. The platform calls the API to query the data in the Hive table based on the table relationships saved when the rule was created, and then processes and displays the data.

[0126] The reconciliation device for JSON data and data warehouse baseline data provided in the embodiments of this application is described below. The reconciliation device for JSON data and data warehouse baseline data described below can be referred to in correspondence with the reconciliation method for JSON data and data warehouse baseline data described above. Figure 2 As shown, this application provides a reconciliation device for JSON data and data warehouse baseline data. The device includes:

[0127] The reconciliation configuration receiving module 201 is used to receive reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse bottom table data, and alarm configurations.

[0128] The reconciliation execution instruction construction module 202 is used to construct reconciliation execution instructions for data comparison based on preset templates and reconciliation mapping relationships;

[0129] The reconciliation result data acquisition module 203 is used to trigger distributed computing resources to load JSON data and data warehouse bottom table data based on reconciliation scheduling information and reconciliation execution instructions, and to perform data comparison and return reconciliation result data.

[0130] The alarm result generation module 204 is used to analyze the reconciliation result data according to the alarm configuration and generate the corresponding alarm results.

[0131] In one embodiment, the reconciliation execution instruction construction module 202 includes:

[0132] The preset template recording unit is used to load preset templates;

[0133] The reconciliation mapping parameter determination unit is used to parse the reconciliation mapping relationship, obtain the reconciliation mapping parameters, and assign the reconciliation mapping parameters to the entity class. The reconciliation mapping parameters include JSON data identifier, data warehouse bottom table data identifier, data filtering conditions, JSON field extraction rules, data warehouse bottom table field concatenation rules, and offset time information.

[0134] The reconciliation execution instruction determination unit is used to replace the corresponding placeholders in the preset template with variables in the entity class to obtain the reconciliation execution instruction for data comparison.

[0135] In one embodiment, the reconciliation mapping parameter determination unit includes:

[0136] The configuration field reading sub-unit is used to read each configuration field in the reconciliation mapping relationship;

[0137] The parameter type identification subunit is used to identify the parameter type of each configuration field based on the field identifier of each configuration field and its position in the configuration structure in the reconciliation mapping relationship;

[0138] The reconciliation mapping parameter determination subunit is used to perform structured parsing on each configuration field based on the parameter type identification result, and to determine the JSON data identifier, data warehouse bottom table data identifier, data filtering conditions, JSON field extraction rules, data warehouse bottom table field concatenation rules, and offset time information to obtain the reconciliation mapping parameters.

[0139] In one embodiment, the reconciliation result data acquisition module 203 includes:

[0140] The reconciliation scheduling information parsing unit is used to parse reconciliation scheduling information and determine the task scheduling parameters used to control the execution of reconciliation tasks.

[0141] The distributed computing task description generation unit is used to encapsulate the reconciliation execution instructions and task scheduling parameters to generate a distributed computing task description.

[0142] The distributed computing task description sending unit is used to send the distributed computing task description to the distributed computing resource through the task submission interface, triggering the distributed computing resource to execute the reconciliation execution instruction according to the task scheduling parameters, so as to load JSON data and data warehouse bottom table data and perform data comparison.

[0143] In one embodiment, the alarm result generation module 204 includes:

[0144] The alarm trigger threshold acquisition unit is used to acquire the alarm trigger threshold in the alarm configuration;

[0145] The reconciliation discrepancy analysis unit is used to analyze the number of reconciliation discrepancies in the reconciliation result data;

[0146] The alarm result generation unit is used to compare the number of reconciliation discrepancies with the alarm trigger threshold and generate alarm results corresponding to the comparison results.

[0147] In one embodiment, the apparatus further includes:

[0148] The chart generation module is used to generate trend charts and detailed tables based on the reconciliation results data;

[0149] The chart display module is used to display trend charts and detailed tables when a result query request is received.

[0150] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the reconciliation method for JSON data and data warehouse bottom table data as described in any of the above embodiments.

[0151] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the reconciliation method for JSON data and data warehouse bottom table data as described in any of the above embodiments.

[0152] Indicatively, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 3 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as applications, that can be executed by the processing component 302. The applications stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the reconciliation methods for JSON data and data warehouse baseline data in any of the above embodiments.

[0153] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0154] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0155] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0156] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0157] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for reconciling JSON data and data warehouse baseline data, characterized in that, The method includes: Receive reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse baseline data, and alarm configurations; Based on the preset template and the reconciliation mapping relationship, construct reconciliation execution instructions for data comparison; Based on the reconciliation scheduling information and the reconciliation execution instruction, the distributed computing resources are triggered to load JSON data and data warehouse bottom table data, and data comparison is performed to return the reconciliation result data; Based on the alarm configuration, the reconciliation result data is analyzed to generate corresponding alarm results.

2. The reconciliation method for JSON data and data warehouse baseline data according to claim 1, characterized in that, The step of constructing a reconciliation execution instruction for data comparison based on a preset template and the reconciliation mapping relationship includes: Load preset template; Parse the reconciliation mapping relationship to obtain the reconciliation mapping parameters, and assign the reconciliation mapping parameters to the entity class. The reconciliation mapping parameters include JSON data identifier, data warehouse base table data identifier, data filtering conditions, JSON field extraction rules, data warehouse base table field concatenation rules, and offset time information. Replace the corresponding placeholders in the preset template with the variables in the entity class to obtain the reconciliation execution instruction for data comparison.

3. The reconciliation method for JSON data and data warehouse baseline data according to claim 2, characterized in that, The step of parsing the reconciliation mapping relationship to obtain the reconciliation mapping parameters includes: Read the configuration fields in the reconciliation mapping relationship; Based on the field identifier of each configuration field and its position in the reconciliation mapping relationship, parameter type identification is performed on each configuration field; Based on the parameter type identification results, each configuration field is structured and parsed to determine the JSON data identifier, data warehouse bottom table data identifier, data filtering conditions, JSON field extraction rules, data warehouse bottom table field concatenation rules, and offset time information, so as to obtain the reconciliation mapping parameters.

4. The reconciliation method for JSON data and data warehouse baseline data according to claim 1, characterized in that, The step of triggering distributed computing resources to load JSON data and data warehouse baseline data based on the reconciliation scheduling information and the reconciliation execution instruction, and performing data comparison, includes: The reconciliation scheduling information is analyzed to determine the task scheduling parameters used to control the execution of the reconciliation task; The reconciliation execution instruction and the task scheduling parameters are encapsulated to generate a distributed computing task description; The distributed computing task description is sent to the distributed computing resource through the task submission interface, triggering the distributed computing resource to execute the reconciliation execution instruction according to the task scheduling parameters, so as to load JSON data and data warehouse bottom table data and perform data comparison.

5. The reconciliation method for JSON data and data warehouse baseline data according to claim 1, characterized in that, The step of analyzing the reconciliation result data and generating corresponding alarm results based on the alarm configuration includes: Obtain the alarm trigger threshold in the alarm configuration; Analyze the number of reconciliation discrepancies in the reconciliation results data; The number of reconciliation discrepancies is compared with the alarm trigger threshold to generate an alarm result corresponding to the comparison result.

6. The reconciliation method for JSON data and data warehouse baseline data according to any one of claims 1 to 5, characterized in that, The method further includes: Based on the reconciliation results data, generate trend charts and detailed tables; When a result query request is received, the trend chart and the detailed table are displayed.

7. A reconciliation device for JSON data and data warehouse baseline data, characterized in that, The device includes: The reconciliation configuration receiving module is used to receive reconciliation scheduling information for controlling task execution, reconciliation mapping relationships for associating JSON data and data warehouse baseline data, and alarm configurations. The reconciliation execution instruction construction module is used to construct reconciliation execution instructions for data comparison based on a preset template and the reconciliation mapping relationship. The reconciliation result data acquisition module is used to trigger distributed computing resources to load JSON data and data warehouse bottom table data based on the reconciliation scheduling information and the reconciliation execution instruction, and to perform data comparison and return reconciliation result data; The alarm result generation module is used to analyze the reconciliation result data according to the alarm configuration and generate corresponding alarm results.

8. The reconciliation device for JSON data and data warehouse baseline data according to claim 7, characterized in that, The reconciliation execution instruction construction module includes: The preset template recording unit is used to load preset templates; The reconciliation mapping parameter determination unit is used to parse the reconciliation mapping relationship, obtain the reconciliation mapping parameters, and assign the reconciliation mapping parameters to the entity class. The reconciliation mapping parameters include JSON data identifier, data warehouse bottom table data identifier, data filtering conditions, JSON field extraction rules, data warehouse bottom table field concatenation rules, and offset time information. The reconciliation execution instruction determination unit is used to replace the corresponding placeholders in the preset template with the variables in the entity class to obtain the reconciliation execution instruction for data comparison.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the reconciliation method for JSON data and data warehouse bottom table data as described in any one of claims 1 to 6.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the reconciliation method for JSON data and data warehouse bottom table data as described in any one of claims 1 to 6.

Images