An offline variable machining platform and device

Abstract

The application relates to the technical field of data processing, in particular to an offline variable processing platform and equipment. The offline variable processing platform comprises a classification node editing module, a script editing module and a batch test module. The classification node editing module is used for creating a plurality of classification nodes. The script editing module is used for creating a structured query statement according to a to-be-queried data source and to-be-queried business data, obtaining a sub-script, and receiving a first target classification node bound by a user for the sub-script. The batch test module is used for receiving a plurality of target sub-scripts selected by the user in a plurality of sub-scripts bound by any second target classification node, and generating a node script according to the target sub-scripts. The node script is used for instructing a third-party big data platform to acquire target business data corresponding to the second target classification node. The batch test module is also used for receiving and saving the target business data. Therefore, the offline variable processing efficiency can be improved, and the error rate is low.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an offline variable processing platform and device. Background Technology

[0002] Offline variables are crucial inputs for risk control decisions. To assess a customer's historical behavioral characteristics (such as pre-loan application behavior, tax records, credit history, etc.), "offline variables" need to be generated based on historical data. These variables are then fed into the approval system or model system as inputs to support risk assessment in scenarios such as loan management, post-loan management, and institutional cooperation.

[0003] In existing technologies, offline variable processing generally relies on manually writing and assembling SQL (Structured Query Language) scripts, and then processing them through code development. This method involves multiple stages and collaboration and communication among multiple departments (such as business, data, and risk control), resulting in a long scheduling period from requirement proposal to production deployment. Overall, it is inefficient and has a high error rate, making it difficult to meet the needs of rapid strategy iteration. Summary of the Invention

[0004] In view of this, embodiments of this application provide an offline variable processing platform and device, which can effectively solve the problems of low efficiency and high error rate in offline variable processing in the prior art.

[0005] In a first aspect, embodiments of this application provide an offline variable processing platform, which includes: a classification node editing module, a script editing module, and a batch testing module;

[0006] The category node editing module is used to create several category nodes;

[0007] The script editing module is used to create a structured query statement based on the data source to be queried and the business data to be queried, to obtain a sub-script, and to receive the first target category node bound by the user to the sub-script;

[0008] The batch testing module is used to receive several target sub-scripts selected by the user, and to generate node scripts based on the several target sub-scripts. The node scripts are used to instruct a third-party big data platform to obtain target business data corresponding to the second target classification node. The several target sub-scripts are selected from multiple sub-scripts bound to any second target classification node.

[0009] The batch testing module is also used to receive and save the target business data.

[0010] In some embodiments, when creating a plurality of the category nodes, the category node editing module is specifically used for:

[0011] The system receives a user-defined node identifier and a primary key field defined for the category node; the primary key field is used to retrieve the business table associated with the target sub-script when generating the node script; wherein the business table contains the primary key field.

[0012] In some embodiments, the script editing module is specifically used for:

[0013] The system receives a first target category node selected by the user, receives several business variables defined by the user for the first target category node based on the business data to be queried, receives a structured query statement edited by the user based on the data source and the several business variables, obtains the sub-script based on the several business variables and the structured query statement, and binds the sub-script to the first target category node.

[0014] In some embodiments, the script editing module is specifically used for:

[0015] Receive the first target category node selected by the user and automatically load the primary key field defined in the first target category node;

[0016] The system receives several business variables defined by the user for the category node based on the business data to be queried; wherein the information of the business variables includes variable identifier, variable type and variable initial value;

[0017] The system receives a query statement from the user for each of the business variables and the primary key field, based on the corresponding target field and data source address within the data source, and assigns the query result to the corresponding variable identifier to obtain the sub-script and bind it to the category node.

[0018] In some embodiments, the script editing module is further configured to: receive master-slave attributes configured by the user for each of the sub-scripts; the master-slave attributes include a master table identifier and a slave table identifier;

[0019] The batch testing module is specifically used for:

[0020] In response to the "Create New Test Task" command, create a new test task;

[0021] Receive a plurality of target sub-scripts selected by the user; wherein the plurality of target sub-scripts are selected from a plurality of sub-scripts bound to any second target classification node;

[0022] Based on the master-slave attributes of each target sub-script, the structured query statements in all the target sub-scripts are concatenated using a left join to obtain the node script;

[0023] The node script is configured in the test task and sent to the third-party big data platform; the third-party big data platform executes the test task to search all relevant data tables based on the primary key field in the node script, and uses structured query statements to retrieve the target business data from each of the data tables.

[0024] In some embodiments, the batch testing module is further configured to: receive a user-defined business data sample and configure the business data sample in the test task;

[0025] The test task is also used to instruct the third-party big data platform to compare the target business data with standard data after obtaining the target business data, obtain test results, and send them back to the batch test module; wherein, the standard data includes the test results of the business data sample or historical node script;

[0026] The batch testing module is also used to: determine whether the test task is successfully tested based on the test results, and if the test task is successfully tested, save the node script based on the version management rules to obtain the final node script, and submit a release application for the final node script.

[0027] In some embodiments, the platform further includes a publishing module;

[0028] The publishing module, in response to the publishing request, generates an online task for the final node script and sends it again to the third-party big data platform;

[0029] The third-party big data platform executes the online task, retrieves the final business data based on the structured query statement in the final node script, and sends it back to the publishing module.

[0030] The publishing module is used to update the stored target business data based on the final business data.

[0031] In some embodiments, the script editing module is further configured to, after verifying that the sub-script is correct, save the sub-script and set the target version number according to the version number management rules to obtain the sub-script with the corresponding target version number;

[0032] The verification includes syntax checking and consistency verification;

[0033] The script editing module is also used to call the detection interface provided by the third-party big data platform before saving the sub-script, to perform syntax detection on the sub-script through the detection interface, and, if the syntax detection passes, to perform consistency verification on the business variables defined in the sub-script and the business variables used in the structured query statement.

[0034] In some embodiments, the platform also provides a query interface, which is used by external systems to query the target business data corresponding to the category node based on the category node identifier.

[0035] Secondly, embodiments of this application provide a terminal device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the functions of various modules in the offline variable processing platform provided in the first aspect of this application.

[0036] The embodiments of this application have the following beneficial effects:

[0037] The offline variable processing platform of this application includes: a category node editing module, a script editing module, and a batch testing module. The category node editing module is used to create several category nodes. The script editing module is used to create structured query statements based on the data source to be queried and the business data to be queried, obtain sub-scripts, and receive the first target category node bound by the user to the sub-scripts. The batch testing module is used to receive several target sub-scripts selected by the user and generate node scripts based on the target sub-scripts. The node scripts are used to instruct a third-party big data platform to obtain the target business data corresponding to a second target category node. The target sub-scripts are selected from multiple sub-scripts bound to any second target category node. The batch testing module is also used to receive and save the target business data. This application can automatically generate node scripts and automatically query the variable values ​​of business variables based on the node scripts, thereby improving the efficiency of offline variable processing and reducing the error rate. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This paper shows a structural block diagram of an offline variable processing platform according to an embodiment of the present application;

[0040] Figure 2 This paper shows a schematic diagram of a classification node editing interface in an offline variable processing platform according to an embodiment of this application;

[0041] Figure 3 This paper shows a schematic diagram of a script editing interface in an offline variable processing platform according to an embodiment of this application;

[0042] Figure 4 Another schematic diagram of the script editing interface in the offline variable processing platform of this application embodiment is shown;

[0043] Figure 5 This paper shows a schematic diagram of a test task configuration interface in an offline variable processing platform according to an embodiment of this application;

[0044] Figure 6 This paper shows a schematic diagram of a batch script selection window in an offline variable processing platform according to an embodiment of this application.

[0045] Figure 7 Another structural block diagram of the offline variable processing platform according to an embodiment of this application is shown.

[0046] Explanation of key component symbols:

[0047] 100 - Offline variable processing platform; 110 - Category node editing module; 120 - Script editing module; 130 - Batch testing module; 140 - Publishing module; 200 - Third-party big data platform; 300 - External systems. Detailed Implementation

[0048] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0049] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0050] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0051] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0052] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0053] In existing technologies, offline variable processing relies on manually writing SQL scripts, which are then assembled and debugged across different environments. The lack of a unified development and management platform results in fragmented processes, low efficiency, and a lack of standardized tools. Therefore, this application provides an offline variable processing platform.

[0054] The main idea of ​​this application is to build a visual, unified, and process-controllable offline variable processing platform to achieve full lifecycle management of offline variables from development, testing to release, thereby improving the efficiency, accuracy and compliance of variable deployment and supporting the rapid iteration of risk control strategies.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0056] The offline variable processing platform will be described below with reference to some specific embodiments.

[0057] Figure 1 A structural block diagram of an offline variable processing platform according to an embodiment of this application is shown. Exemplarily, the offline variable processing platform 100 includes: a classification node editing module 110, a script editing module 120, and a batch testing module 130.

[0058] The offline variable processing platform 100 communicates with a third-party big data platform 200, and also communicates with a business system.

[0059] The category node editing module 110 is used to create several category nodes.

[0060] In this embodiment, different classification nodes are created based on different business categories. These classification nodes are used to categorize and manage subsequently defined business variables (such as personal credit information, tax information, and loan information), forming a standardized data asset catalog and improving data reuse and management efficiency.

[0061] The script editing module 120 is used to create a structured query statement based on the data source and the business data to be queried, obtain sub-scripts based on the structured query statement, and receive the first target category node that the user binds to the sub-script. The first target category node can be any category node that the sub-script wants to bind to. Multiple sub-scripts can be bound to a single category node.

[0062] As an example, in this application embodiment, different sub-scripts are created based on different data sources or business categories due to different query data sources or different business categories, and preset business data in the data source is obtained through the sub-scripts.

[0063] The batch testing module 130 is used to receive several target sub-scripts selected by the user, and to generate node scripts based on the several target sub-scripts. The several target sub-scripts are selected from among the multiple sub-scripts bound to any second target category node. The node scripts are used to instruct the third-party big data platform 200 to obtain the target business data corresponding to the second target category node. The second target category node can be any category node, but is used here only to distinguish it from the first target category node. In other words, the batch testing module is used to receive several target sub-scripts selected by the user from among the multiple sub-scripts bound to any second target category node.

[0064] After creating the sub-scripts, this embodiment also supports functional testing of the scripts. When testing scripts in batches, multiple sub-scripts under the same category node are generally selected. The selected target sub-scripts are then merged to obtain the node script.

[0065] The third-party big data platform 200 can retrieve target business data based on structured query statements within node scripts. For example, it can retrieve fields such as the unified social credit code (custno), company number (companyno), company name (companyname), and company ID (companyid) from tables stored within the third-party big data platform 200. It is understandable that third-party big data platforms (such as the Ark platform) mainly include the following functions: big data ecosystem integration, natively supporting metadata collection from components such as Hadoop (distributed computing framework), Hive (data warehouse system), HBase (distributed columnar database), Kafka (distributed message queue system), and Spark (distributed in-memory computing engine); metadata management, unifying the registration of data assets such as tables, fields, and indicators; data classification and tagging, labeling sensitive data (such as corporate credit codes) with tags such as "sensitive personal information" and "financial data," supporting compliance control and access control; category search and discovery, supporting the search of all related tables by field name, improving data availability and automation; security and access control integration, working with Ranger (data security management framework) and Sentry (access control component) to achieve fine-grained access control, ensuring the secure output of results after left join.

[0066] The batch testing module 130 is also used to receive and save target business data.

[0067] In one implementation, the category node editing module 110, when creating several category nodes, is specifically used for:

[0068] It receives the user-defined node identifier and the primary key field defined for the category node; the primary key field is used to retrieve the business table associated with the target sub-script when generating the node script; the business table contains the primary key field.

[0069] The node identifiers mentioned above include, but are not limited to, node names. The primary key fields mentioned above include, but are not limited to, customer number, company number, and unified social credit code.

[0070] Exemplary, the platform in this application embodiment includes a visual editing interface for editing category nodes through a category node editing interface.

[0071] like Figure 2As shown, the category node editing module 110 receives the node's English name, Chinese name, primary key field, Chinese name of the primary key field, table name of the data to be queried for business data, main query parameters, input strategy set, update frequency, product application, and whether it is enabled, all created by the user through the category node editing interface. For example, a category node is a defined credit reporting object (credit reporter, individual or company), such as the primary key field being the company number (companyno).

[0072] It is important to note that the primary key field and node identifier cannot be changed once created, as they will be used as unique identifiers thereafter.

[0073] Furthermore, the script editing module 120 is specifically used for:

[0074] The script editing module 120 receives the first target category node selected by the user, several business variables defined by the user for the first target category node based on the business data to be queried, and a structured query statement edited by the user based on the data source and the business variables. It then generates a sub-script based on the business variables and the structured query statement and binds the sub-script to the first target category node. A category node can be bound to multiple sub-scripts. Specifically, the script editing module 120 is also used for: automatically loading primary key fields.

[0075] Exemplary, the platform in this application embodiment includes a script editing interface; such as Figure 3 As shown, the script editing interface includes a script information editing section and a query statement editing section. Figure 3 In this context, "Hdp_loanbank_ysodst.testjianfeng_20241111" is a database table name.

[0076] The script editing module 120 receives the first target category node to be bound selected by the user through the script information editing section, receives several business variables defined by the user for the category node using the script information editing section based on the business data to be queried, and obtains variable information based on the several business variables; it also receives the structured query statement edited by the user using the query statement editing section based on the data source and several business variables, obtains the sub-script based on the variable information and the structured query statement, and binds both the variable information and the sub-script to the first target category node.

[0077] This application centrally edits and stores business variables through the script information editing module, and edits the SQL logic corresponding to each business variable based on the query statement editing module, supporting script addition, deletion, modification, query and historical tracing.

[0078] Furthermore, the script editing module 120 is specifically used for:

[0079] S211 receives the first target category node selected by the user and automatically loads the primary key field defined within the first target category node. For example... Figure 3 As shown, the script information editing module receives the user's first target category node and loads and displays the primary key field of the first target category node.

[0080] S212, Receive several business variables defined by the user for the category node based on the business data to be queried; wherein, the information of the business variables includes variable identifier, variable type, and variable initial value. Variable identifier includes, but is not limited to, variable name. For example... Figure 3 As shown, users can create and delete business variables using the script information editing section based on the business data to be queried. This section includes function keys for creating and deleting business variables, as well as controls for configuring variable names, Chinese names, types, default values ​​(initial values), and descriptions. For example, business variables could include customer name, company name, loan date, and the company's unified social credit code.

[0081] S213 receives the user's query statement for each business variable and primary key field, based on the corresponding target field and data source address within the data source. The query results are then assigned to the corresponding variable identifiers, resulting in a sub-script that is bound to the category node. Users can edit the structured query statement for the business variables to be queried using the structured query statement editing section, based on structured query statement keywords, and set aliases for the business variables. Structured query statement keywords include select, from, etc.

[0082] Furthermore, the script editing module 120 is also used to: receive the master-slave attributes configured by the user for each sub-script; the master-slave attributes include the master table identifier and the slave table identifier. For example... Figure 4 As shown, you can select a master-slave attribute for each sub-script so that the sub-script identified by the slave table can be merged into the sub-script identified by the master table in subsequent sub-scripts.

[0083] In one implementation, the batch testing module 130 is specifically used to perform the following steps:

[0084] S311, in response to the New Test Task command, creates a new test task.

[0085] The platform in this embodiment of the application has a built-in scheduling system, such as the Dolphin Scheduler. Users trigger test task commands through the test interface, and the batch test module 130 creates test tasks through the interface provided by the Dolphin Scheduler.

[0086] like Figure 5 As shown, the platform in this embodiment includes a test task configuration interface; users can select the corresponding sub-scripts of the test task, batch test types (new or modified scripts), etc. through the test task configuration interface.

[0087] S312, receive a plurality of target sub-scripts selected by the user, wherein the plurality of target sub-scripts are selected from a plurality of sub-scripts bound to any second target classification node.

[0088] The second target category node is any category node, and each category node is bound to several sub-scripts.

[0089] like Figure 6 As shown, the user selects a target sub-script within the batch test script selection window included in the test task configuration interface. The batch test script selection window displays multiple sub-script entries, each including a category node, sub-script identifier, script version, and a corresponding selection button. The sub-script identifier includes, but is not limited to, the sub-script name.

[0090] S313. Based on the master-slave attributes of each target sub-script, concatenate the structured query statements in all target sub-scripts using a left join to obtain the node script.

[0091] The script editing module 120 concatenates subscripts identified by the master table with master-detail attributes, as well as subscripts identified by the slave table with master-detail attributes, using a left join to obtain a node script. For example, a node script includes the business variables of all target subscripts, the first subscript, the second subscript, the join condition of the left join, etc., concatenated together using a left join. Then the statement structure of this node script is:

[0092] Select

[0093] All business variables

[0094] from(

[0095] select

[0096] Business variables within the first sub-script

[0097] from

[0098] Database table address )

[0100] left join(

[0101] Select

[0102] Business variables within the first sub-script

[0103] from

[0104] Database table address

[0105] The connection condition for a left join.

[0106] The join condition of a left join includes the primary key field. The purpose of the join condition of a left join is to tell the database to join the left and right tables according to the set primary key field. As long as the primary key field in the left table is equal to the primary key field in the right table, they will be merged into one row of output.

[0107] Understandably, based on the master-slave attributes and the set primary key field of each target sub-script, the structured query statements in all target sub-scripts are concatenated using a left join to obtain the node script.

[0108] For example, create a category node for a credit reporting object, then create a first sub-script to retrieve the credit reporting object's identity information, and a second sub-script to retrieve the credit reporting object's loan information. When editing the first sub-script, create business variables such as the credit reporting object's name, ID number, and address in the script information editing section, and retrieve the primary key field (e.g., the ID number). In the query statement editing section, edit the structured query statement (SQL statement) to retrieve the credit reporting object's name, ID number, and address. Configure the first sub-script's master-slave attribute as the master table identifier. When editing the second sub-script, create business variables such as the credit reporting object's name, ID number, lending bank name, and loan amount in the script information editing section, and retrieve the primary key field (e.g., the ID number). In the query statement editing section, edit the structured query statement to retrieve the credit reporting object's name, lending bank name, and loan amount. Configure the second sub-script's master-slave attribute as the slave table identifier. Then, the user selects the category node corresponding to the credit reporting object on the test task configuration interface, and then selects the first and second sub-scripts. Finally, the first subscript is joined to the second subscript using a left join.

[0109] Understandably, the node script is a JSON data structure. It includes multiple data category nodes, each serving as a top-level key to identify a set of business attributes. For example, "Node A" contains the first type of business variables, while "Node B" contains the second type. Each category node contains a nested object, which consists of several key-value pairs representing the variable name and value of a specific business variable.

[0110] S314, configure the node script in the test task and send it to the third-party big data platform 200; the third-party big data platform 200 executes the test task, searches all relevant data tables based on the primary key field in the node script, and uses structured query statements to retrieve the target business data from each data table.

[0111] To ensure stability, the script editing interface also provides a lock button for each sub-script, allowing only the previous user to edit that sub-script, preventing other users from editing it. When a user clicks the lock button on any sub-script, a lock command is triggered. The script editing module 120 responds to this lock command by locking the sub-script, preventing other users from accessing or opening it again. The script editing module 120 responds to an unlock command to release the lock.

[0112] Furthermore, to ensure the robustness and correctness of the platform, the script editing module 120 is also used to save the sub-script after verifying that the sub-script is correct, and set the target version number according to the version number management rules to obtain the sub-script with the corresponding target version number.

[0113] The verification includes syntax checking and consistency verification; it is understood that in the embodiments of this application, the same query function includes multiple versions of sub-scripts.

[0114] The script editing module 120 is also used to call the detection interface provided by the third-party big data platform 200 before saving the sub-script, to perform syntax detection on the sub-script through the detection interface, and to perform consistency verification on the business variables defined in the sub-script and the business variables used in the structured query statement if the syntax detection passes.

[0115] Syntax checking includes keyword checking for structured query statements, business variable field checking, data source checking, and data table checking. Keyword checking detects missing or incorrect keywords; business variable field checking detects spelling errors in business variables; data source checking checks for the existence of the data source and spelling errors; and data table checking detects the existence of the table and spelling errors.

[0116] Consistency verification is used to compare whether the defined business variables and the business variables in the structured query statement are consistent, thereby discovering potential data discrepancies and improving robustness.

[0117] In one implementation, the batch testing module 130 is further configured to: receive a user-defined business data sample and configure the business data sample in the test task.

[0118] The test task is also used to instruct the third-party big data platform 200 to compare the target business data with the standard data after obtaining the target business data, obtain the test results, and send them back to the batch test module 130; wherein, the standard data includes test results of business data samples or historical node scripts; historical node scripts are the node scripts of the same category node names in the past.

[0119] Understandably, the target business data includes the actual variable values ​​corresponding to the business variables, and the business data samples include the theoretical sample values ​​corresponding to the business variables. The test results of the historical node scripts include the historical variable values ​​of the business variables, and these historical variable values ​​have passed the tests.

[0120] In one implementation, a third-party big data platform 200 compares the target business data with sample business data to check whether the actual variable value of each business variable matches the theoretical sample value. If they match, the business variable passes the test; otherwise, the test fails. A simplified table of test results is shown in Table 1.

[0121] Table 1. Summary of Test Results

[0122]

[0123] The third-party big data platform 200 is also used to summarize and statistically analyze the testing status of each business variable and the final test pass rate to obtain the test results. It calculates the percentage of all business variables that passed the test and compares this percentage with a set pass rate threshold. If the percentage is greater than the pass rate threshold, the test task is considered passed; otherwise, the test task fails. For example, if the pass rate threshold is set to 90%, the test task is considered passed only if 90% or more of the actual variable values ​​of the business variables match the theoretical sample values. If the consistency percentage is less than 50%, the test task fails. In other words, the node script test fails. As an example, the summary of the test results is shown in Table 2.

[0124] Table 2 Summary of Test Results

[0125]

[0126] In another implementation, the third-party big data platform 200 compares the target business data with the test results of historical node scripts (hereinafter referred to as historical node results) to check whether the actual variable value of each business variable is consistent with the historical variable value of that business variable. If they are consistent, the business variable passes the test; otherwise, the test fails. For example, the test results include test task details, which include the primary key field, category node, historical node results, current node result (i.e., the test result for this time), business data sample, whether the variable passed (whether the business variable passed the test), whether there is a business data sample, whether the node script has test results, whether the node script passed the test, creation time, etc., as shown in Table 3.

[0127] Table 3 Test Task Details

[0128]

[0129] The batch testing module 130 is also used to determine whether a test task is successful based on the test results. If the test task is successful, it saves the node script according to version management rules to obtain the final node script, and submits a release request for the final node script. After the test task passes, formal data querying is performed.

[0130] Test results include the status of whether the test task passed or failed, the reason for failure, and the success or failure status of each business variable. The batch testing module 130 is also used to display test results based on the test results.

[0131] This application's embodiments introduce a mandatory testing and verification mechanism to ensure deployment quality. Before being released to the production environment, it must undergo a "batch testing" process: it supports configuring business data samples corresponding to sub-scripts for batch testing; it automatically executes SQL and generates field-level test task details and statistical test results; only after the test tasks pass can it enter the deployment process, effectively preventing erroneous logic from being deployed and ensuring data accuracy.

[0132] Furthermore, such as Figure 7 As shown, the platform also includes a publishing module 140.

[0133] The publishing module 140, in response to the publishing request, generates a deployment task for the final node script and sends it again to the third-party big data platform 200. The platform in this embodiment also includes a publishing interface with a publishing button. The user operates the publishing button to trigger a publishing request. In response to the publishing request, the publishing module 140 generates a deployment task by calling the scheduling system's interface and sends the deployment task to the third-party big data platform 200.

[0134] The third-party big data platform 200 executes the online task, retrieves the final business data based on the structured query statement in the final node script, and sends it back to the publishing module 140.

[0135] The publishing module 140 is used to update the target business data based on the final business data.

[0136] The platform in this application embodiment also includes an approval module. This module receives deployment applications, receives approval instructions from users based on the deployment applications, and sends the deployment task to a third-party big data platform 200 based on the approval instructions. This ensures that the operation is error-free or that there is no risk of malicious modification.

[0137] Furthermore, users need to apply for access permissions from the third-party big data platform 200 based on their account on the offline variable processing platform 100. When executing test and deployment tasks, the third-party big data platform 200 also needs to verify whether the offline variable processing platform 100 has permission to access big data source tables such as Hive, to avoid task failure due to permission issues. This facilitates permission and asset management.

[0138] This application embodiment can integrate automated publishing and scheduling to ensure a closed-loop process. After the test is passed and the review is completed, the platform automatically calls the publishing interface of the downstream system (such as the "Ark Platform"), submits information such as task name, complete script, execution time, and write-back table name, and updates the scheduling task (such as Dolphin Scheduling), triggering the execution of the offline processing node script, and finally obtaining the final target business data.

[0139] In one implementation, to facilitate viewing of business variable information, the platform also includes an input item information module; the input item information module corresponds to the input item interface.

[0140] The input information module is used to retrieve all category nodes, automatically retrieve all business variables within all sub-scripts bound to each category node, and display all business variables of each category node.

[0141] The input information module is used to display information related to each business variable on the input interface. The information related to the business variable includes the name of the category node bound to the sub-script, the Chinese name of the category node, the name of the sub-script to which it belongs, the variable type of the business variable, the variable value, the configuration strategy set, the version of the category node, etc.

[0142] In one implementation, the platform also provides a query interface for external systems 300 to query target business data corresponding to category nodes based on category node identifiers. External systems 300 include, but are not limited to, audit systems or RFS systems. The RFS system is a system that processes and summarizes business variables, returning the summarized variables to the approval system to invoke the decision engine system. For example, external system 300 could be an RFS system. Processed offline variables can be queried externally through the RFS system (based on key fields such as unified social credit code and ID card), allowing the approval system to call and access the decision engine, thus enabling rapid transformation of analysis results into policy applications.

[0143] This application adopts a unified platform for visual operation to replace decentralized manual development. It changes the previous situation of relying on manual writing and splicing of SQL scripts, multi-stage collaboration, and lack of unified tool support. The offline variable processing is centralized on a visual platform, and users can directly write SQL scripts through the page interface, reducing the technical threshold and error risk.

[0144] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the offline variable processing platform 100 described above.

[0145] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0146] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0147] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0149] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0150] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An offline variable processing platform, characterized in that, The offline variable processing platform includes: a classification node editing module, a script editing module, and a batch testing module; The category node editing module is used to create several category nodes for category management based on the received data created by the user through the category node editing interface, including the node's English name, Chinese name, primary key field, Chinese name of the primary key field, table name of the data to be queried for business data storage, main query parameters, input strategy set, update frequency, product application, and whether it is enabled. The primary key field is used to retrieve the business table associated with the target sub-script when generating the node script; wherein, the business table contains the primary key field. The script editing module is used to create a structured query statement based on the data source to be queried and the business data to be queried, to obtain a sub-script, and to receive the first target category node bound by the user to the sub-script; and to receive the master-slave attributes configured by the user for each sub-script; the master-slave attributes include a master table identifier and a slave table identifier; The batch testing module is used to create new test tasks in response to the command to create new test tasks. Receive a plurality of target sub-scripts selected by the user; wherein the plurality of target sub-scripts are selected from a plurality of sub-scripts bound to any second target classification node; Based on the master-detail attributes of each target sub-script and the primary key field contained in the business table associated with each target sub-script, the structured query statements in all target sub-scripts are concatenated using a left join to obtain the node script; The node script is configured in the test task and sent to a third-party big data platform; the third-party big data platform executes the test task to search all relevant data tables based on the primary key field in the node script, and uses structured query statements to retrieve the target business data from each of the data tables; The batch testing module is also used to receive and save the target business data.

2. The offline variable processing platform according to claim 1, characterized in that, The script editing module is specifically used for: The system receives a first target category node selected by the user, receives several business variables defined by the user for the first target category node based on the business data to be queried, receives a structured query statement edited by the user based on the data source and the several business variables, obtains the sub-script based on the several business variables and the structured query statement, and binds the sub-script to the first target category node.

3. The offline variable processing platform according to claim 2, characterized in that, The script editing module is specifically used for: Receive the first target category node selected by the user and automatically load the primary key field defined in the first target category node; The system receives several business variables defined by the user for the first target category node based on the business data to be queried; wherein the information of the business variables includes variable identifier, variable type and variable initial value; The user receives a query statement for each of the business variables and the primary key field, based on the corresponding target field and data source address within the data source, and assigns the query result to the corresponding variable identifier, thereby obtaining the sub-script and binding it to the category node.

4. The offline variable processing platform according to claim 1, characterized in that, The batch testing module is also used to: receive a business data sample set by the user, and configure the business data sample in the test task; The test task is also used to instruct the third-party big data platform to compare the target business data with standard data after obtaining the target business data, obtain test results, and send them back to the batch test module; wherein, the standard data includes the test results of the business data sample or historical node script; The batch testing module is also used to: determine whether the test task is successfully tested based on the test results, and if the test task is successfully tested, save the node script based on the version management rules to obtain the final node script, and submit a release application for the final node script.

5. The offline variable processing platform according to claim 1, characterized in that, The platform also includes a publishing module; The publishing module, in response to the publishing request, generates an online task for the final node script and sends it again to the third-party big data platform; The third-party big data platform executes the online task, retrieves the final business data based on the structured query statement in the final node script, and sends it back to the publishing module. The publishing module is used to update the stored target business data based on the final business data.

6. The offline variable processing platform according to claim 1, characterized in that, The script editing module is also used to save the sub-script after verifying that the sub-script is correct, and set the target version number according to the version number management rules to obtain the sub-script with the corresponding target version number; The verification includes syntax checking and consistency verification; The script editing module is also used to call the detection interface provided by the third-party big data platform before saving the sub-script, to perform syntax detection on the sub-script through the detection interface, and, if the syntax detection passes, to perform consistency verification on the business variables defined in the sub-script and the business variables used in the structured query statement.

7. The offline variable processing platform according to claim 1, characterized in that, The platform also provides a query interface, which allows external systems to query the target business data corresponding to the category node based on the category node identifier.

8. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the functions of each module in the offline variable processing platform according to any one of claims 1-7.

Images