An intelligent data integration and variable derivation system

Abstract

The application discloses an intelligent data integration and variable derivation system, relates to the technical field of data processing and integration, and provides a highly customizable data model definition through a visual configuration module; the core innovation lies in that a dynamic script execution engine is introduced, interface authentication and processing logic online editing and instant effect are supported, and publishing can be completed without restarting the service; a unique two-dimensional array expression structure is adopted in the variable derivation engine, real-time derivation calculation of complex variables is realized through recursive reference, the variable mapping work of a business system is greatly simplified, and the statistical performance is guaranteed through the timing task and table separation strategy of a data monitoring and statistics module; as a whole, the system improves the data integration docking efficiency from days to minutes, reduces complex variable processing from several days to ten minutes, significantly reduces the development and operation cost, and has strong error recovery, intelligent cache and flexible charging capabilities.

Description

Technical Field

[0001] This invention relates to the field of data processing and integration technology, specifically to an intelligent data integration and variable derivation system. Background Technology

[0002] In traditional data integration and variable derivation scenarios, business systems typically need to interface with numerous heterogeneous external data sources (such as third-party APIs, databases, and files). This process faces several challenges: First, each data source has different interface protocols, authentication methods, data formats, and calling specifications, requiring developers to write hard-coded calling, parsing, and error handling logic for each interface, resulting in long development cycles, low code reusability, and high maintenance costs. Second, the variables required by the business often do not come directly from the original data but require complex logical operations and combinations for derivation. This logic is usually scattered across various business codebases, making it difficult to manage, reuse, and dynamically adjust in a unified manner. Furthermore, to improve system availability and performance, it is often necessary to manually implement common mechanisms such as interface call routing, failure retries, result caching, and frequency control. These repetitive tasks distract the development of core business logic. In addition, when interface logic or business rules need to be changed, it is often necessary to modify the source code, retest, and redeploy the entire application, making rapid response and hot updates impossible.

[0003] Therefore, the industry urgently needs an intelligent data integration solution that can be uniformly configured, efficiently executed, flexibly expanded, and support real-time variable derivation to solve the problems of low development efficiency, poor flexibility, and difficult maintenance. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent data integration and variable derivation system that enables efficient connection and integration of multi-source data, including data from third-party platforms, file systems, and database systems. It not only enables cross-platform and cross-system data interoperability and sharing, but also has a powerful data variable derivation capability. It can automatically extract basic variables, derived variables, and script variables from the connected data to meet the data needs of different business systems and solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an intelligent data integration and variable derivation system, comprising:

[0006] The visualization configuration module is configured to provide a human-computer interaction interface for building standardized data models for different external data sources. The data model includes at least the definitions of interface access parameters, authentication strategies, and returned data structures.

[0007] The dynamic script execution engine, connected to the visualization configuration module, is configured to parse and execute preset logic scripts in real time during system operation to complete pre-processing, post-processing, and dynamic authentication of data source requests. The logic scripts support online editing and immediate effect.

[0008] A variable derivation engine is configured to generate derived variables based on the original data returned by the data source. It includes an operational variable processing unit, which is configured to calculate results by parsing hierarchical reference expressions. The hierarchical reference expressions allow the current calculation item to reference the calculation results of the previous calculation item as input parameters.

[0009] The intelligent routing and scheduling module is configured to manage call requests to the external data source according to a preset routing strategy, and to perform failover when a call anomaly is detected.

[0010] Preferably, the computational variable processing unit uses a multidimensional array structure to store the hierarchical reference expression, where each sub-item in the multidimensional array is an independent calculation formula or expression array.

[0011] Preferably, the method for the computational variable processing unit to calculate the target index variable value includes the following steps:

[0012] Recursively traverse the sub-item corresponding to the target index and all its referenced preceding sub-items;

[0013] Flatten the sub-items into a one-dimensional expression sequence;

[0014] Parse the variable reference identifiers in the one-dimensional expression sequence and replace them with actual values;

[0015] The expression result is calculated according to the preset operation priority rules.

[0016] Preferably, the dynamic script execution engine is built using a script language interpreter that supports dynamic compilation. It is configured to hot update the logic script by reloading the class definition or script object without interrupting the existing service process when a script change instruction is received. The application scenarios of the logic script at least cover: signature generation of interface request parameters, decryption processing of return messages, and cleaning logic for non-standardized data.

[0017] Preferably, the intelligent routing and scheduling module includes a manual routing mode and an automatic routing mode;

[0018] In the manual routing mode, priority identifiers and call weight ratios are configured for different data sources of the same data dimension;

[0019] The intelligent routing and scheduling module is configured to prioritize calling high-priority data sources, and only automatically downgrade to calling low-priority data sources if all the highest-priority data sources experience call anomalies or timeouts.

[0020] Preferably, the visualization configuration module also includes a multi-dimensional billing management unit;

[0021] The multi-dimensional billing management unit, in the billing lookup mode, parses the interface return message and determines whether the data is valid based on user-defined lookup rules. The lookup rules support Boolean operations, numerical comparisons, character matching, and enumeration of logical combinations to judge the returned fields. Billing logic is triggered only when a lookup rule is matched.

[0022] As a preferred option, it also includes a data monitoring and statistics module, configured to periodically execute statistical logic using scheduled tasks and persist the statistical results to a summary table, adopt a time sharding strategy to manage the underlying data tables, and be configured to automatically initialize the transaction table structure for future time windows through scheduled tasks, and restrict historical detail query operations that cross preset time spans to ensure system response performance.

[0023] Preferably, the interface access parameters in the data model include a retry strategy configuration, which includes a timeout threshold and a number of retry attempts. If no response is received within the timeout threshold, the call is determined to have failed, and a new call is automatically initiated based on the number of retry attempts until the call succeeds or the maximum number of retry attempts is reached.

[0024] Preferably, the visualization configuration module also includes an interface testing and simulation unit. The interface testing and simulation unit is configured to allow users to directly initiate real interface calls in the configuration interface and display the original return messages. It supports user-defined settings for return sample data and is used for internal system logic verification when external interfaces are unavailable.

[0025] Preferably, the computational variable processing unit is configured to process all intermediate calculation results using a high-precision numerical type when performing numerical calculations, and to truncate the final output according to a preset precision strategy. It is also configured to directly determine that the calculation result of the current expression is invalid when it detects that the numerical value involved in the calculation is a missing value and no missing value filling strategy is configured.

[0026] In summary, the beneficial effects of this invention are:

[0027] This invention achieves a high degree of automation and flexible customization of data integration and variable derivation processes through the synergistic effect of a visual configuration module, a dynamic script execution engine, and a variable derivation engine. This significantly improves the response efficiency and data processing capabilities of business systems. Specifically, the system supports online editing and instant-on Groovy scripts, enabling interface integration, authentication logic adjustments, and variable derivation rule modifications to be completed without restarting the service, reducing data integration time from days in traditional development models to minutes. Simultaneously, the variable derivation engine, based on a multidimensional array expression structure, supports recursive parsing and real-time calculation of complex formulas, drastically reducing the workload of variable mapping and processing for complex interfaces from the original 3 days to less than 10 minutes, greatly reducing the complexity and time cost of business system integration. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the operational framework of an intelligent data integration and variable derivation system according to the present invention;

[0030] Figure 2 This is a schematic diagram of the system construction framework of an intelligent data integration and variable derivation system according to the present invention;

[0031] Figure 3 This is a schematic diagram of the engine layer operation process of an intelligent data integration and variable derivation system according to the present invention;

[0032] Figure 4 This is a schematic diagram of the test interface of an intelligent data integration and variable derivation system according to the present invention;

[0033] Figure 5 This is a schematic diagram of the query and billing interface of an intelligent data integration and variable derivation system according to the present invention;

[0034] Figure 6 This is a schematic diagram of the stepped configuration interface of an intelligent data integration and variable derivation system according to the present invention;

[0035] Figure 7 This is a schematic diagram of the billing configuration interface of an intelligent data integration and variable derivation system according to the present invention;

[0036] Figure 8 This is a schematic diagram of the script management interface of an intelligent data integration and variable derivation system according to the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0038] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.

[0039] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0040] Any feature disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by other equivalent or similar features for a similar purpose, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0041] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; they can refer to the internal communication of at least two elements or the interaction relationship of at least two elements, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0042] Please see Figures 1-8 The present invention provides an embodiment of an intelligent data integration and variable derivation system, comprising a visual configuration module, a dynamic script execution engine, a variable derivation engine, an intelligent routing and scheduling module, and a data monitoring and statistics module, wherein:

[0043] The visualization configuration module is configured to provide a human-computer interaction interface for building standardized data models for different external data sources. The data models excel in high configurability and customization, robust error handling and recovery capabilities, intelligent caching and update strategies, flexible billing and call frequency control, ease of integration and expansion, and comprehensive security and access management. These outstanding performances are based on a well-designed data model, whose key attributes / fields include:

[0044] 1. Interface address: URL request address;

[0045] 2. Interface access method: Request method can be GET or POST;

[0046] 3. Interface authentication script: pre-processor and post-processor;

[0047] 4. Timeout field: Configures the timeout period for API calls. The timeout period can be selected as 1 / 3 / 5 / 10 / 15 / 20 / 30 / 60 / 120 seconds.

[0048] 5. Number of retries: The number of times to retry after a failed API call. The number of retries can be selected as 1 / 2 / 3.

[0049] 6. Header fields: Customizable request header fields;

[0050] 7. Cookies field: Customizable Cookies field information;

[0051] 8. Interface parameter types: body format, form-data format, URL parameter, allowing users to directly initiate real interface calls in the configuration interface and display the original return message, supporting user-defined settings for return sample data, used for internal system logic verification when external interfaces are unavailable;

[0052] The HTTP message body format: Data is placed in the HTTP message body. In the system, it specifically refers to `content-Type: application / json;` or `application / xml;`.

[0053] The `form-Data` format specifies POST form data. Specifically, it refers to `content-Type: form-data` or `x-www-form-urlencoded`.

[0054] URL format: URL parameters, such as ?a=xxx&b=xxx; also called queryString;

[0055] 9. Request parameter configuration: You can select basic variables and constants;

[0056] 10. Non-null value judgment: single field judgment and condition group judgment;

[0057] 11. Interface call type: synchronous call, asynchronous call;

[0058] 12. Data return type: JSON, XML;

[0059] 13. Data return sample: can be customized;

[0060] 14. API call test: After providing the configuration, access the real API to verify whether the configuration is correct;

[0061] 15. Result caching configuration: minutes, hours, days;

[0062] 16. Status code retrieval path;

[0063] 17. Success status code field;

[0064] 18. Status information path;

[0065] 19. Call frequency limits: minutes, hours, days, months;

[0066] 20. Disabled period: Hour;

[0067] 21. Billing Mode Configuration: This feature allows for billing query, billing retrieval, and periodic billing. Users can customize unit price, billing period, free usage, and other information. When billing retrieval is supported, the interface returns a message, and the validity of the data is determined based on user-defined query rules. These query rules support Boolean operations, numerical comparisons, character matching, and enumeration of logical combinations to judge the returned fields. Billing logic is triggered only when a query rule is matched.

[0068] A dynamic script execution engine, connected to the visualization configuration module, is configured to parse and execute preset logic scripts in real time while the system is running. This completes pre-processing, post-processing, and dynamic authentication of data source requests. The logic scripts support online editing and immediate effect. Specifically, they are built using a scripting language interpreter that supports dynamic compilation, such as Groovy. We have innovatively introduced Groovy, a move that greatly improves the system's flexibility and efficiency. Groovy, with its powerful dynamic compilation and reloading features, enables us to implement a series of remarkable functions, the most prominent of which is the ability to edit online and publish instantly.

[0069] With Groovy, we've successfully broken free from many constraints of traditional development processes. Previously, whenever we needed to integrate with new third-party data interfaces, modify authentication scripts for those interfaces, or adjust variable derivation logic, development teams often had to go through tedious coding, testing, deployment, and restart processes. Now, however, all of that is completely different.

[0070] Groovy's dynamic compilation capabilities allow us to edit scripts online and apply changes instantly without interrupting system operation. This means that whether adding new data interfaces, updating authentication policies, or optimizing variable derivation logic, we can complete these changes quickly and apply them to the system immediately. This capability not only significantly shortens the development cycle and reduces operational costs but also enables us to respond more flexibly to market changes and customer needs.

[0071] Furthermore, Groovy's real-time variable derivation capabilities have brought unprecedented convenience to our business. It allows us to dynamically calculate and generate new variables based on real-time data while the system is running, thus greatly enriching our data processing and analysis capabilities. This feature plays a crucial role in data processing, real-time monitoring, and decision support, providing more accurate and timely data support for our business decisions.

[0072] The introduction of Groovy not only improved the technical level and performance of our business systems, but also brought us unprecedented flexibility and efficiency. It enables us to respond more quickly to market changes and meet demands.

[0073] The variable derivation engine is configured to generate derived variables based on the raw data returned by the data source. It includes a computational variable processing unit and uses a multidimensional array structure to store the hierarchical reference expressions. Each item in the multidimensional array is an independent calculation formula or expression array. Specifically, the computational variables support both basic and complex variable derivation. This technology solves the complexity of variable derivation and reduces the time-consuming and labor-intensive variable mapping and processing issues during business system integration. For complex interfaces, the workload has been reduced from three days to less than ten minutes.

[0074] Our computational variable calculation system is built upon a two-dimensional array, where each item can be an expression or a formula that includes references to other items. This design allows us to construct complex computational formulas and solve for the final result through recursion and variable filling.

[0075] Core Structure

[0076] Two-dimensional array: Each element can be an array of expressions (such as ["(","6","+","9",")","÷","5"]) or a formula that references other sub-items (such as ["{{0}}","÷","{{1}}"], where "{{0}}" and "{{1}}" refer to the 0th and 1st sub-items in the two-dimensional array, respectively).

[0077] The calculation process is as follows:

[0078] 1. Tile expression:

[0079] For a given output index (e.g., output=2), we recursively flatten the sub-item corresponding to that index and all the sub-items it references into a one-dimensional expression array.

[0080] For example, for output=2, the flattened expression array is ["(","(","6","+","9",")",÷","5",")","÷","(","1","+","{var}",")"].

[0081] Convert this one-dimensional array into a pure string expression, such as =((6+9)÷5)÷(1+var), where {var} is the variable to be filled.

[0082] 2. Fill in the variable values:

[0083] Iterate through the array of one-dimensional expressions, replacing variables (such as {var}) with their actual values ​​(such as 5).

[0084] For example, if var=5, the expression becomes =((6+9)÷5)÷(1+5).

[0085] 3. Calculate the expression:

[0086] Start by evaluating the expression from the innermost set of parentheses.

[0087] Use mathematical operation rules, such as multiplication and division followed by addition and subtraction, or more advanced methods to calculate the value of each subexpression.

[0088] The calculation results are filled back into the expression array to form a new expression array.

[0089] For example, first calculate (6+9)÷5=3, then calculate 1+5=6, and finally calculate 3÷6=0.5. In actual calculations, BigDecimal should be used to maintain precision.

[0090] Note: During the calculation process, all numerical operations are handled using the BigDecimal class to ensure precision and round to a maximum of 10 decimal places.

[0091] If the data is null, a global missing value strategy is used to fill it, such as using 0 or a specific missing value.

[0092] If any value is used in the calculation with null (except for the nvl method), the result is null.

[0093] If the denominator is 0, the result is null.

[0094] Example as follows:

[0095] Input: Two-dimensional array = [["(","6","+","9",")","÷","5"],["1","+","{var}"],["{{0}}","÷","{{1}}"]]; var=5; output=2.

[0096] Output: After the above calculation steps, the final result is 0.5, or an approximate value obtained according to the rounding rules.

[0097] The intelligent routing and scheduling module is configured to manage call requests to the external data source according to a preset routing strategy, and to perform failover when a call anomaly is detected, including manual routing mode and automatic routing mode;

[0098] In the manual routing mode, priority identifiers and call weight ratios are configured for different data sources of the same data dimension;

[0099] The intelligent routing and scheduling module is configured to prioritize calling high-priority data sources, and only automatically downgrade to calling low-priority data sources if all the highest-priority data sources experience call anomalies or timeouts.

[0100] The data monitoring and statistics module is configured to periodically execute statistical logic using scheduled tasks and persist the statistical results to a summary table. It adopts a time sharding strategy to manage the underlying data tables and is configured to automatically initialize the transaction table structure for future time windows through scheduled tasks, and restrict historical detail query operations that cross preset time spans to ensure system response performance.

[0101] Specifically, in optimizing the response speed of statistical business systems, we have adopted a series of innovative and efficient strategies, the most important of which is the ingenious integration of scheduled task functionality and database partitioning design.

[0102] First, we fully utilized the scheduling module provided by the spring-context package in the Spring framework to build a powerful scheduled task system. Through a carefully designed scheduled thread pool, we can periodically trigger and execute a series of preset statistical tasks. After these tasks run efficiently, they accurately persist the statistical results to the summary statistics table. This not only ensures the real-time nature and accuracy of the data but also keeps the query speed of related report data stably controlled at the second level, providing users with an extremely smooth user experience.

[0103] Meanwhile, to address the large volume of log records generated during data integration, we adopted a proactive solution: a date-based table partitioning strategy. Through scheduled tasks, we automatically initialized the partitioning structure for the next three months for all transaction tables at the beginning of each month. This design not only effectively avoided performance bottlenecks caused by data expansion but also made data management and querying clearer and more efficient.

[0104] Of course, data partitioning also brings certain limitations. To maintain system responsiveness, we implemented time-based queries for historical details, prohibiting queries across months. While this limitation somewhat affects user flexibility, the performance improvement it brings is obvious. Over time, the system's response speed has consistently remained at the millisecond level, providing users with stable and efficient service.

[0105] By cleverly utilizing scheduled task functionality and database partitioning design, we successfully optimized the performance of statistical business systems, bringing users a smoother and more efficient user experience.

[0106] It is worth mentioning that in this embodiment, we introduced the Groovy scripting language, and Python is also available as an option, which has some advantages in data computation.

[0107] For real-time data statistics, we use a scheduled approach combined with table partitioning and a real-time data warehouse for rapid analysis, reducing the pressure on the business system itself. However, choosing a real-time data warehouse is more expensive.

[0108] The scheduled task component can be selected from more third-party dependencies such as xxl-job, but we used the one that comes with the framework itself, which reduced the complexity of the project.

[0109] In summary, the advantages of this invention are as follows:

[0110] High configurability and customization:

[0111] This data model offers a wealth of configuration options, allowing users to highly customize various aspects of interface integration according to specific needs. Whether it's interface parameters, header information, cookies, authentication scripts, or failure retry strategies, users can flexibly configure them according to their actual requirements.

[0112] This high degree of configurability allows the model to be applied to a variety of different application scenarios and third-party data interfaces, improving the model's versatility and practicality.

[0113] Powerful error handling and recovery capabilities:

[0114] The model has a built-in retry mechanism, which automatically retryes multiple times when an API call fails, until it succeeds or the set retry limit is reached. This helps to handle abnormal situations such as network instability and API failure, improving the reliability and stability of the API.

[0115] In addition, the model also supports path configuration for the returned status codes and status information, enabling users to more accurately judge the results of interface calls and take corresponding error handling measures.

[0116] Intelligent caching and update strategies:

[0117] The model supports result caching configuration, allowing you to set the cache time based on the frequency of API usage and data update cycle. This helps reduce unnecessary API calls and improves system response speed and performance.

[0118] Meanwhile, the model also provides an intelligent cache update strategy. When cached data expires or needs to be updated, it can automatically trigger a new interface call to ensure the real-time performance and accuracy of the data.

[0119] Flexible billing and call frequency control:

[0120] The model supports multiple billing configurations, including query billing, lookup-based billing, and periodic billing, allowing users to choose the appropriate billing method based on their actual needs. This helps reduce the cost of using the interface and improves economic efficiency.

[0121] In addition, the model also provides call frequency limit configuration, which can set the upper limit of the number of calls within different time units according to actual needs. This helps to avoid excessive calls to the interface and protect the stability and security of third-party data interfaces.

[0122] Easy to integrate and extend:

[0123] This data model employs a standardized interface design, making integration with other systems simpler and more convenient. Whether integrating with internal systems, third-party systems, or cloud platforms, it can be achieved through simple configuration and interface docking.

[0124] Furthermore, the model offers rich extension interfaces and plugin mechanisms, allowing users to add new features or extend existing ones according to their actual needs. This helps maintain the model's advanced nature and adaptability, meeting ever-changing market demands.

[0125] Comprehensive security and access control:

[0126] The model incorporates pre-processors and post-processors to implement secure operations such as authentication, encryption, and decryption of the interface. This helps protect the data security and privacy of the interface, preventing unauthorized access and tampering.

[0127] Furthermore, the model supports the verification and management of the identity and permissions of API callers, ensuring that only legitimate users can access and use the API. This helps maintain the security and stability of the system.

[0128] Ease of use:

[0129] Visual Configuration: The model provides a visual configuration interface, allowing users to complete interface integration configuration through simple clicks and drag-and-drop operations. This significantly lowers the technical barrier to interface integration, enabling even non-technical personnel to easily get started.

[0130] Customizable Data Return Samples: The model supports custom modification of the returned data samples, allowing users to set the format and content of the returned data according to their actual needs. This helps to better meet the needs of real-world application scenarios and improves the ease of use and flexibility of the interface.

[0131] Status code and status information path configuration: The model supports configuring status code retrieval paths and status information paths, allowing users to set these paths according to the actual situation of the interface. This helps to more accurately determine the success or failure of the interface call, improving the reliability and stability of the interface.

[0132] Timeliness of business response:

[0133] Thanks to the robust data model and the dynamic features and reloading capabilities of the Groovy language, business development teams have seen a significant improvement in their efficiency in responding to requirements, reducing data integration and docking time from days to minutes.

[0134] In practice, taking the creation of data integration and the generation of basic variables as an example, the operation steps are as follows:

[0135] 1. Enter the interface management in data access, add and fill in the basic information of the data integration interface. The data type can be selected from JSON or XML. Select the data dimension that has been established. Name and unique identifier are required fields. Data type and data dimension are required. Remarks are optional fields. The interface type is synchronous interface by default, but can be switched to asynchronous interface.

[0136] 2. Data integration interface request configuration. In the request configuration section, you can select GET or POST from the drop-down menu. POST is the default.

[0137] After selecting the request method, enter the request address in the input box below. The interface request address can be up to 200 characters long.

[0138] The authentication script can be selected from the drop-down menu of existing interface integration scripts in the script management section. Clicking the jump button allows you to jump to the script management page to add a new script.

[0139] The timeout period can be selected from 1 / 3 / 5 / 10 / 15 / 20 / 30 / 60 / 120 seconds, and is left blank by default.

[0140] The number of retries can be selected as 1 / 2 / 3 times. The default is not selected. It is not required. If not selected, it is considered as 0 retries.

[0141] When the data platform accesses an external interface, if no response is received within the specified timeout period, the request will be resent according to the set number of retries. If the interface still has not received a response after the retries have expired, it is considered a call exception, and the transaction will be handled as an exception.

[0142] The request content can switch between Body parameters, Form-Data parameters, URL parameters, Headers, and Cookies.

[0143] Note: Body = Data is placed in the HTTP message body. In the system, it specifically refers to content-Type: application / json; or application / xml;

[0144] `form-Data` refers to POST form data. Specifically, it refers to `content-Type form-data` or `x-www-form-urlencoded`.

[0145] URL: URL parameters, such as ?a=xxx&b=xxx; also called queryString;

[0146] headers: Request headers;

[0147] Cookies: A special request header.

[0148] The key, description, type, and value in the request parameters should be configured according to the actual requirements of the interface. The type can be selected from the dropdown menu as either a constant or a variable. If the type is selected as a variable, you need to choose a pre-configured input parameter variable from the system. Enter the test value to be requested and select whether the line is empty. Clicking "Add Parameter Configuration" will add a new line of parameter configuration; you can also delete lines by clicking the delete button next to each line.

[0149] 3. After completing the configuration in the request configuration section, click the test button. The output results will return the response message from the API call. Refer to [link / reference]. Figure 4 ;

[0150] 4. After clicking "Test", the test output results are displayed on the left. At the same time, the same message as the test output results on the left is displayed in the custom results column on the right. You can click "Set Custom Results" to customize the results according to your needs. Clicking "Set Custom Message" will bring up a pop-up window to edit the message.

[0151] 5. Status Parameter Configuration: Enter the cache time in the input field of the status parameter configuration section. The value must be a positive integer less than 30. The unit can be selected as minutes, hours, or days. The default is no caching if no input is entered. Cache is optional. The status configuration displays the path to the status code returned by the interface, the code value of the returned fields, and the field path of the status information. Fields in the interface test return message can be directly selected. The call limit supports setting the upper limit of calls by minutes, hours, days, and months. The time starts from the first call. The call time limit supports selecting a prohibited call time period, supporting multiple time periods.

[0152] 6. Multiple interface billing types include query billing, search billing, and time billing. Tiered and frequency billing supports both search and query billing, and users can configure unit price, total price, tiered calculation, time period, etc.

[0153] Select "Query Billing," and a billing configuration window will pop up below. The billing period is required. Select the desired calculation type (per-use billing / tiered billing). To query billing, only the unit price needs to be entered; the unit price supports integer and floating-point numbers. Trial rules can also be set. Single or multiple selections are allowed. Single selection allows you to choose the trial expiration date or the number of trials; multiple selections mean the trial ends when either condition is met. Trial settings are optional.

[0154] The trial expiration date can be selected directly from the calendar. The trial count input field now supports integer data types. (See reference.) Figure 5 ;

[0155] Billing is based on a fixed unit price and the number of calls. Select "Query Billing" or "Find-Based Billing" as the billing type, choose the corresponding billing period, and then select "Billing by Call." A unit price input box will pop up on the page; enter the unit price. "Find-Based Billing" will verify that the query rules have been entered; otherwise, it cannot be saved. The backend will calculate the total query amount based on the number of API calls and the unit price.

[0156] Tiered billing calculates charges based on the number of calls within a given tier price period. It supports setting a unit price and the number of calls within each tier, and allows for resetting the number of calls within a tier period by refreshing the tier period.

[0157] Select "Billing Inquiry" / "Billing Query" for the billing type, choose the corresponding billing period, and select "Tiered Billing". For "Billing Query", you need to fill in the query rules. After selecting "Tiered Billing", the tiered configuration box will expand on the page.

[0158] Upon entering the page, a single line of tiered configuration is displayed by default. Enter the frequency range and the unit price for that range. Click the plus sign to create a multi-level tier. Click delete to delete a tier. Deletion is not possible if there is only one line; the message "Cannot delete more" will appear. By default, the number of times starts from 1 when configuring a tier, which is fixed by the system. After entering the number of times in the input boxes, clicking "Add Tier" will display the tiers below starting from the next higher tier with 1 more times. All preceding times are not supported as they are strongly associated with the parent tier. If the customer needs to exclude intermediate ranges from the tier, simply enter 0 for the unit price.

[0159] You can also choose whether to use a tiered cycle to reset the number of times it expires. When you click on the tiered cycle, a tiered cycle drop-down box will pop up. There are three default types: natural month, natural quarter, and natural year. It also prompts: When you cross to the next tiered cycle, the number of tiered times will be reset to zero.

[0160] The number of times input field in the tiered configuration supports integer type, while the unit price input field supports both floating-point and integer types. All values ​​must be positive. (See reference...) Figure 6 .

[0161] Check the billing details for reference. Figure 7 ;

[0162] The query-based billing adds the configuration of query rules. Users can manually configure query rules according to the query conditions, and billing will be applied when the query conditions are met.

[0163] The query rules support multi-line condition selection. Users select fields from the interface's returned message based on the query conditions for calculation. The interaction for selecting fields in the path is as follows: Click "Please Select," and a pop-up window will appear displaying the message from the interface output. Select a specific field.

[0164] After selecting the rule field to be retrieved, choose the operation method: greater than, equal to, less than, greater than or equal to, less than or equal to, not equal to, contain, do not contain, hit enumeration, miss enumeration, contain prefix, do not contain prefix, empty, not empty, empty string, not empty string. There is no character length limit for the input result.

[0165] Multi-line fields support adding single conditions and condition groups. Single conditions are as described above. Condition groups present two rows of fields for selection, allowing users to add or delete multiple rows. Condition group rules are divided into "and" and "or". Clicking the "X" deletes all configuration content within the field. When only one field configuration remains, "and" or "or" will no longer be displayed, regardless of whether they were previously displayed before the field.

[0166] It supports rule-based judgments on multiple conditions, including all conditions being met, any one condition being met, none of the conditions being met, and at least one condition being met.

[0167] Periodic billing: Periodic billing is charged once based on the selected time period, with no limit on the number of calls within the period, and is a package billing method.

[0168] Select the "Periodic Billing" type, enter the total price, and choose the specific billing period (a period of time from the two time options). You can also set trial rules. The total price input box supports floating-point and integer formats.

[0169] For interfaces whose billing is about to expire, send a reminder message every day for 5 days before the expiration date. Send a message at 8:00 AM every day: "Your billing period is about to expire. Please take note!"

[0170] Trial rules can also be set, and these rules are universal.

[0171] 7. Create a script, refer to Figure 8 ;

[0172] Script Name: Limited to 50 characters, required field;

[0173] Unique Identifier: Limited to 50 characters, required field;

[0174] Data dimensions: The system has already established the classification dimensions;

[0175] Script type: Interface integration script, variable derivation script, service output script. Default is not filled in, but it is required.

[0176] Note: Entries within 200 characters are not required.

[0177] The script configuration can be edited in the script editor, or imported by uploading a Groovy script file. The system will automatically parse the content of the uploaded script into the input box on the front end.

[0178] 8. Create variables;

[0179] Variable name: Limited to 50 characters, required;

[0180] Unique Identifier: Limited to 50 characters, required field;

[0181] Data dimensions: The system has already established the classification dimensions;

[0182] Data type: Select from dropdown menus: character, integer, floating-point, boolean, object, array, date;

[0183] Data source: Select from dropdown menus: Interface type, File type, SQL type, DB configuration type, DB-SQL type;

[0184] Note: Entries within 200 characters are not required.

[0185] When the source of the variable is an interface, an interface variable configuration pop-up will appear, requiring the variable to be mapped to a specific output field in an interface;

[0186] The system supports different routing methods, including manual routing and automatic routing.

[0187] Manual routing allows users to automatically adjust the priority and weight of interfaces, influencing the selection of interfaces.

[0188] Automatic routing relies entirely on the system to make automated judgments and selections.

[0189] The "Interface" option allows you to select an interface that is already configured and available in the data platform. When selecting an interface, you can only select the same type of interface, and you can only select synchronous interfaces or asynchronous interfaces. When the first synchronous interface is selected, the selection of new interfaces will be filtered so that only synchronous interfaces can be selected. The same applies to asynchronous interfaces.

[0190] The "Interface Output" option displays a detailed output message hierarchy in a pop-up window, including reference values ​​for each field, and allows selection of the returned fields. The pop-up window should also support keyword searching within the message.

[0191] "Priority" indicates the priority of a variable calling different interfaces; the smaller the number, the higher the priority.

[0192] A priority level can contain multiple interfaces. Lower priority interfaces will only be invoked if all interfaces with the highest priority level encounter exceptions. Priority levels can be added, removed, moved up, or moved down.

[0193] The "call ratio" is used to determine the probability of an interface being called, and its effect is limited to interfaces within the same priority level. The sum of the call ratios for interfaces within the same priority level must equal 100; otherwise, the system will report an error and the changes will not be saved.

[0194] If all interfaces time out or report errors, the system will handle the exceptions.

[0195] Intelligent routing eliminates the need to set priorities or call ratios. All interfaces are at the same level, and all decision-making logic can be implemented in the backend. Simply configure the interface output fields, and the backend will automatically recognize the interface information to achieve intelligent routing. Finally, the output value is configured.

[0196] The above description is merely a specific embodiment of the invention, but the scope of protection of the invention is not limited thereto. Any variations or substitutions conceived without inventive effort should be included within the scope of protection of the invention. Therefore, the scope of protection of the invention should be determined by the scope defined in the claims.

Claims

1. An intelligent data integration and variable derivation system, characterized in that: include: The visualization configuration module is configured to provide a human-computer interaction interface for building standardized data models for different external data sources. The data model includes at least the definitions of interface access parameters, authentication strategies, and returned data structures. The dynamic script execution engine, connected to the visualization configuration module, is configured to parse and execute preset logic scripts in real time while the system is running. It uses the standardized data model to complete the pre-processing, post-processing, and dynamic authentication of data source requests. The logic scripts support online editing and immediate effect. A variable derivation engine is configured to generate derived variables based on the original data returned by the data source. It includes a computational variable processing unit configured to calculate results by parsing hierarchical reference expressions. These hierarchical reference expressions allow the current calculation item to reference the calculation results of its predecessors as input parameters. The computational variable processing unit stores these hierarchical reference expressions using a two-dimensional array structure. Each item in the two-dimensional array is an independent calculation formula or an array of expressions that references other items. When calculating the value of a target index variable, the computational variable processing unit recursively traverses the item corresponding to the target index and all its referenced predecessors, flattens the traversed items into a one-dimensional expression sequence, parses the variable reference identifiers in the one-dimensional expression sequence and replaces them with the corresponding actual values, and then calculates the expression result according to a preset computational priority rule. The intelligent routing and scheduling module is configured to manage call requests to the external data source according to a preset routing strategy, and to perform failover when a call anomaly is detected.

2. The intelligent data integration and variable derivation system according to claim 1, characterized in that: The dynamic script execution engine is built using a script language interpreter that supports dynamic compilation. It is configured to hot update the logic script by reloading the class definition or script object without interrupting the existing service process when a script change instruction is received. The application scenarios of the logic script at least cover: signature generation of interface request parameters, decryption processing of return messages, and cleaning logic for non-standardized data.

3. The intelligent data integration and variable derivation system according to claim 2, characterized in that: The intelligent routing and scheduling module includes a manual routing mode and an automatic routing mode; In the manual routing mode, priority identifiers and call weight ratios are configured for different data sources of the same data dimension; The intelligent routing and scheduling module is configured to prioritize calling high-priority data sources, and only automatically downgrade to calling low-priority data sources if all the highest-priority data sources experience call anomalies or timeouts.

4. The intelligent data integration and variable derivation system according to claim 3, characterized in that: The visualization configuration module also includes a multi-dimensional billing management unit; The multi-dimensional billing management unit, in the billing lookup mode, parses the interface return message and determines whether the data is valid based on user-defined lookup rules. The lookup rules support Boolean operations, numerical comparisons, character matching, and enumeration of logical combinations to judge the returned fields. Billing logic is triggered only when a lookup rule is matched.

5. The intelligent data integration and variable derivation system according to claim 4, characterized in that: It also includes a data monitoring and statistics module, configured to periodically execute statistical logic using scheduled tasks and persist the statistical results to a summary table. It adopts a time sharding strategy to manage the underlying data tables and is configured to automatically initialize the transaction table structure for future time windows through scheduled tasks, and restrict historical detail query operations that cross preset time spans to ensure system response performance.

6. The intelligent data integration and variable derivation system according to claim 5, characterized in that: The interface access parameters in the data model include a retry strategy configuration, which includes a timeout threshold and a number of retry attempts. If no response is received within the timeout threshold, the call is determined to have failed, and a new call is automatically initiated based on the number of retry attempts until the call succeeds or the maximum number of retry attempts is reached.

7. The intelligent data integration and variable derivation system according to claim 6, characterized in that: The visualization configuration module also includes an interface testing and simulation unit. The interface testing and simulation unit is configured to allow users to directly initiate real interface calls in the configuration interface and display the original return messages. It supports user-defined settings for return sample data and is used for internal system logic verification when external interfaces are unavailable.

8. The intelligent data integration and variable derivation system according to claim 7, characterized in that: When performing numerical calculations, the computational variable processing unit is configured to process all intermediate calculation results using a high-precision numerical type and truncate them according to a preset precision strategy in the final output. It is also configured to directly determine that the calculation result of the current expression is invalid when it detects that the numerical value involved in the calculation is a missing value and no missing value filling strategy is configured.

Images