Data construction method and related device
By generating input interfaces for custom variables and creating script actions, the problem of data construction in complex business scenarios in existing technologies is solved, realizing the flexibility and universality of data construction rules and adapting to the differences in development and testing environments.
Patent Information
- Application Number
- CN202410611211.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-16
- Publication Date
- 2025-11-18
AI Technical Summary
Existing data construction methods are difficult to understand and modify in complex business scenarios, resulting in high difficulty in writing and modifying script code and difficulty in adapting to differences in development and testing environments.
By acquiring script instructions based on the target scripting language, input interfaces for custom variables are generated, and script actions are created based on the script instructions. These actions are then added to the data construction rules, and the script actions are executed to adapt to the needs of different environments.
It improves the flexibility and versatility of data construction, realizes the reusability and environment independence of script actions, and reduces the complexity of data construction tasks.
Smart Images
Figure CN120973353A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data construction method and related apparatus. Background Technology
[0002] Data construction refers to combining data elements according to specific formats and algorithms to construct data with clear meaning. Current data construction methods involve code engineers on a data construction team writing corresponding scripts based on the data construction requirements, uploading the code to a code hosting platform for unified management, and then downloading and running the code from a local office computer or remote machine to complete the data construction task.
[0003] While the above methods can accomplish basic data construction tasks, the different code execution environments during development and testing, as well as the differences in code structure written by different people, make it difficult for other members of the team to understand and use these script codes. Therefore, when faced with complex data construction logic, the writing and modification of script codes is quite difficult. Summary of the Invention
[0004] This application provides a data construction method and related apparatus, which solves the problem that existing data construction platforms are unable to implement data construction in complex business scenarios.
[0005] One aspect of this application provides a data construction method, comprising:
[0006] Retrieves script instructions based on the target scripting language, including custom variables represented by preset characters;
[0007] The corresponding input interface is generated based on the custom variable, and the input interface is used to obtain the variable value of the custom variable;
[0008] Create script actions based on script instructions;
[0009] Add script actions to data construction rules. Data construction rules include execution actions for at least one step. The script actions correspond to the target step in the data construction rule.
[0010] When executing the target step of the data construction rule, the script action is executed based on the variable value.
[0011] In one possible implementation, obtaining script instructions based on the target scripting language includes:
[0012] In response to the action creation operation, the action creation interface is displayed, which includes a script editing window;
[0013] Obtain script commands through the script editing window.
[0014] In one possible implementation, the action creation interface also includes an optional script library;
[0015] After displaying the action creation interface in response to the action creation operation, it also includes:
[0016] Obtain reference information for the target script library. The target script library is an optional script library. The optional script library includes the target algorithm, and the script instructions include the execution string that calls the target algorithm.
[0017] When executing the target step of the data construction rule, script actions are performed based on variable values, including:
[0018] When executing the target step of the data construction rule, the target algorithm is invoked to execute script actions based on the variable values.
[0019] In one possible implementation, before displaying the action creation interface in response to the action creation request, the following is also included:
[0020] In response to the installation operation of the target script library, the target script library is installed using a package management tool corresponding to the target script language;
[0021] Add the target script library as an optional script library.
[0022] In one possible implementation, before obtaining the reference information to the target script library, which is an optional script library, the following steps are also included:
[0023] In response to a version modification operation on the target script library, update the target script library to the specified version via the code repository.
[0024] In one possible implementation, script actions are added to the data construction rules, including:
[0025] In response to the rule creation operation of data construction rules, the rule creation interface is displayed, which includes a step addition window;
[0026] Add the script action to the step addition window that corresponds to the target step.
[0027] In one possible implementation, after executing the script action based on the variable values during the target step of executing the data construction rule, the following is also included:
[0028] Generate and execute runtime logs corresponding to data construction rules; the runtime logs include script instructions.
[0029] Output runtime logs.
[0030] In one possible implementation, before executing script actions based on variable values during the target step of executing the data construction rules, the following is also included:
[0031] In response to the execution of the data construction rules, return the input interface;
[0032] Obtain variable values through the input interface.
[0033] Another aspect of this application provides a data construction apparatus, comprising:
[0034] The acquisition module is used to acquire script instructions based on the target scripting language. The script instructions include custom variables represented by preset characters.
[0035] The generation module is used to generate corresponding input interfaces based on custom variables. The input interfaces are used to obtain the variable values of the custom variables.
[0036] Create a module for creating script actions based on script instructions;
[0037] The acquisition module is also used to add script actions to data construction rules. The data construction rules include execution actions for at least one step, and the script actions correspond to the target steps in the data construction rules.
[0038] The execution module is used to perform script actions based on variable values when executing the target steps of the data construction rules.
[0039] In one possible implementation,
[0040] The acquisition module is specifically used to respond to the action creation operation, display the action creation interface, which includes a script editing window; and to acquire script instructions through the script editing window.
[0041] In one possible implementation, the action creation interface also includes an optional script library;
[0042] The device also includes:
[0043] The acquisition module is also used to acquire reference information to the target script library. The target script library is an optional script library, which includes the target algorithm. The script instructions include the execution string that calls the target algorithm.
[0044] The execution module is specifically used to call the target algorithm to execute script actions based on variable values when executing the target steps of the data construction rules.
[0045] In one possible implementation, the device further includes:
[0046] The installation module is used to respond to the installation operation of the target script library by installing the target script library through the package management tool corresponding to the target script language; and by adding the target script library as an optional script library.
[0047] In one possible implementation, the device further includes:
[0048] The update module is used to respond to version modification operations on the target script library by updating the target script library to the specified version via the code repository.
[0049] In one possible implementation,
[0050] The acquisition module is specifically used to respond to the rule creation operation in response to the data construction rules, and to display the rule creation interface, which includes a step addition window; and to add script actions to the step addition window corresponding to the target step.
[0051] In one possible implementation, the device further includes:
[0052] The log module is used to generate runtime logs corresponding to the executed data construction rules. The runtime logs include script instructions; the runtime logs are also output.
[0053] In one possible implementation, the device further includes:
[0054] The input module is used to respond to the execution of data construction rules and return an input interface; variable values can be obtained through the input interface.
[0055] Another aspect of this application provides a computer device, including: a memory and a processor;
[0056] The memory stores instructions, which, when executed on the processor, perform the methods described above.
[0057] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.
[0058] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.
[0059] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:
[0060] This application provides a data construction method and related apparatus. First, a script instruction is obtained, which is written in a target scripting language and includes custom variables represented by preset characters. By searching the preset characters, the custom variable can be determined, thereby generating a corresponding input interface so that the user can input different variable values as needed. Furthermore, after obtaining the script instruction, script actions can be created based on the script instruction. Then, a data construction rule is obtained, which can consist of one or more steps, each step corresponding to an execution action, wherein the created script action corresponds to the target step. When executing the target step of the data construction rule, the script action is executed based on the variable values input by the user. After executing all steps of the data construction rule, data construction is achieved.
[0061] The method provided in this application, after obtaining the script instructions, creates a corresponding input interface based on the custom variables represented by preset characters, so that users can input variable values, since the script instructions include custom variables. It also creates script actions based on the script instructions, which can be added to the data construction rules as a step in the data construction rules, so as to achieve the effect of flexibly adjusting the data construction rules. When executing the data construction rules, the script instructions can be executed according to the variable values input by the user, so as to adapt to the data requirements of the development environment and the testing environment.
[0062] Therefore, the data construction method provided in this application improves the flexibility of data construction by adding script actions to the data construction rules. It also improves the versatility of data construction by setting custom variable values so that the constructed data construction rules can be applied to both development and testing environments. This satisfies different data construction needs and achieves the effect of reusable and maintainable construction capabilities (script actions). It solves the problem that existing data construction platforms are unable to implement data construction in complex business scenarios. Attached Figure Description
[0063] Figure 1 This is a flowchart of the data construction method provided in the embodiments of this application;
[0064] Figure 2 This is a flowchart of the data construction method provided in the embodiments of this application;
[0065] Figure 3 A schematic diagram of the action creation interface in the data construction method provided in the embodiments of this application;
[0066] Figure 4 This is a flowchart of the pip update process in the data construction method provided in the embodiments of this application;
[0067] Figure 5A schematic diagram of the rule creation interface in the data construction method provided in the embodiments of this application;
[0068] Figure 6 This is a schematic diagram of the running log file in the data construction method provided in the embodiments of this application;
[0069] Figure 7 A schematic diagram of the data construction architecture provided in the embodiments of this application;
[0070] Figure 8 This is a flowchart illustrating the execution of the data construction rules provided in the embodiments of this application.
[0071] Figure 9 This is a schematic diagram of the database table design in the data construction rules provided in the embodiments of this application;
[0072] Figure 10 This is a schematic diagram of the data construction apparatus provided in the embodiments of this application;
[0073] Figure 11 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation
[0074] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0075] Data construction refers to combining data elements according to specific formats and algorithms to construct data with clear meaning. The data construction process is typically handled by a dedicated data construction team. Code engineers write corresponding scripts based on the data construction requirements. These scripts are usually written in various programming languages, such as Python, Java, or C++. After completing the scripts, the code engineers upload them to a code hosting platform (such as Git or SVN) for unified management. Once uploaded, team members can access the code via their local work computers or remote machines. For example, they can log into their code hosting platform accounts and download the required code to their local environment. Then, they configure the appropriate development environment locally and run the code to complete the data construction task.
[0076] While the above methods can accomplish basic data construction tasks, the different code execution environments during development and testing, as well as the differences in code structure written by different personnel, may make it difficult for other members of the team to understand and use these script codes. In scenarios involving high data security requirements, such as the construction of merchant numbers in cross-border remittances and the construction of order numbers in payment order scenarios, complex data construction logic needs to be addressed. In these cases, relying solely on basic code management platforms makes the writing and modification of script codes quite difficult.
[0077] To reduce the difficulty and complexity of data construction tasks and improve the flexibility and reusability of code, this application provides a data construction method and related apparatus. The method involves creating script actions based on script instructions and adding these actions to the data construction rules. Furthermore, after obtaining the script instructions, a corresponding input interface is generated based on custom variables within the script instructions. This allows users to input corresponding variable values according to their environment (development or testing). When executing the steps corresponding to the script actions in the data construction rules, the user can execute the corresponding script actions using the variable values, thereby completing the data construction in the current environment. This method can be applied to different environments according to user needs, thus achieving flexible allocation of data construction rules and improving the flexibility and reusability of the script instructions.
[0078] For ease of understanding, the data construction method and related apparatus provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. First, please refer to... Figure 1 , Figure 1 A flowchart of the data construction method provided in this application embodiment includes:
[0079] 101. Obtain script instructions based on the target scripting language, including custom variables represented by preset characters.
[0080] It is understood that in the embodiments of this application, the data construction method can be implemented through one or more script actions, each script action corresponding to one or more steps in the data construction. To create a script action, it is first necessary to obtain script instructions, which can be script code directly entered by the user or ready-made script code obtained from other sources. These script instructions are written based on a target scripting language. It is understood that scripting languages can include Python, JavaScript, Ruby, PHP, Perl, Groovy, Lua, Bash, etc., and the target scripting language can be any one of these; no limitation is imposed here. It is understood that different scripting languages correspond to different script libraries and package management tools, and these scripting languages have unique uses and advantages in different fields and environments. The choice of which scripting language to use can be made according to specific needs and scenarios.
[0081] The script instructions include custom variables represented by preset characters (such as "$", "%", "@", etc.). For example, if the script instructions include "${list_id}", it means that the custom variable is list_id. This means that the corresponding script action can be executed by obtaining the value of the variable list_id, thereby realizing the data construction task corresponding to the current environment requirements.
[0082] 102. Generate corresponding input interfaces based on custom variables. The input interfaces are used to obtain the variable values of the custom variables.
[0083] Understandably, custom variables can be identified using preset characters in script instructions, and corresponding input interfaces can be generated to retrieve variable values. For example, after determining the custom variable `list_id` based on the preset character "$" in the script instructions, a text box (i.e., a form of input interface) corresponding to `list_id` is displayed on the corresponding interface. Users can provide specific values for the custom variable through the text box, increasing the script's flexibility and reusability.
[0084] 103. Create script actions based on script instructions.
[0085] Understandably, corresponding script actions can be created based on the script instructions obtained in step 101 above. These actions may include function calls, conditional statements, loops, etc. Script actions are used to execute the corresponding scripts and define how various data are processed or transformed during the data construction process.
[0086] 104. Add script actions to data construction rules. Data construction rules include execution actions for at least one step. Script actions correspond to the target step in the data construction rule.
[0087] Understandably, data construction rules, or the logic of data construction, refer to a set of steps that define how to generate or transform data from one or more data sources. This includes one or more actions executed sequentially, with each action corresponding to a step. These steps may include data cleaning, transformation, aggregation, population, generating new data, etc. The action may be a script action generated through script instructions, or it may be other actions.
[0088] Adding a script action to a data construction rule is equivalent to introducing a custom script into the data construction process. This script will be executed at a certain stage of the data construction, meaning a specific step in the data construction rule is required to execute the script action; this step is the target step. This script could be a Python function, a shell script, an SQL script, or any other executable code snippet.
[0089] By embedding script actions into data construction rules, the system can more easily manage and execute complex data processing flows.
[0090] 105. When executing the target step of the data construction rule, execute script actions based on variable values.
[0091] After adding the script action to the data construction rule, during the execution of the data construction rule, when the target step is reached, the above script action will be executed based on the obtained variable value.
[0092] It's understandable that the variable value input interface can be set in either the interface for obtaining script instructions or the interface for executing data construction rules. That is, after responding to the user's action of executing a data construction rule, a text input box for the custom variable (input interface) is displayed. After the user inputs and confirms the custom variable, the data construction rule is executed. When executing the data construction rule, script instructions can be executed based on the variable values input by the user, thus adapting to the data requirements of both development and testing environments.
[0093] The method provided in this application, after obtaining script instructions, creates corresponding input interfaces based on the custom variables represented by preset characters, since the script instructions include custom variables, so that users can input variable values. Based on these variable values, script actions are dynamically executed, solving the complexity of configuration management and maintenance caused by different variable values in development and testing environments. This allows the constructed data construction rules to be applied to both development and testing environments, achieving environment independence and improving the versatility of data construction. Furthermore, script actions are created based on the script instructions. These script actions can be added to the data construction rules as a step within the rules. Both the script actions and the data construction rules can be saved and reused, reducing the need to repeatedly write the same logic and improving the flexibility of data construction. By decomposing the data processing flow into a series of steps and actions, the data processing flow can be more easily managed and adjusted to meet different data construction needs, achieving reusable and maintainable construction capabilities (script actions). This solves the problem that existing data construction platforms struggle to implement data construction in complex business scenarios.
[0094] In this application Figure 1 In one alternative embodiment of the data construction method provided in the corresponding implementation, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the data construction method provided in an embodiment of this application.
[0095] 201. In response to the action creation operation, display the action creation interface, which includes a script editing window and an optional script library.
[0096] 202. Obtain script instructions through the script editing window. Script instructions include custom variables represented by preset characters.
[0097] It is understandable that steps 201 and 202 correspond to... Figure 1 Step 101 in the illustrated embodiment.
[0098] This application embodiment can be applied to a data construction platform, which can be a software development tool, a web application, or a standalone software application. On this data construction platform, a user can initiate an operation to create a new data construction action. This operation can be triggered by a button, menu item, or other user interface element. When the user clicks or selects the action creation operation, the data construction platform displays an action creation interface. This interface allows the user to define a new execution action, specifically a script action created based on script instructions.
[0099] The action creation interface can include a script editing window, specifically a text box, for entering script instructions to define the behavior of the action. The script instructions include custom variables represented by preset characters, allowing for subsequent location of these custom variables based on the preset characters.
[0100] 203. Obtain reference information for the target script library. The target script library is an optional script library. The optional script library includes the target algorithm, and the script instructions include the execution string that calls the target algorithm.
[0101] In this embodiment of the application, the action creation interface also includes an optional script library, which may include one or more script libraries for the user to select. Each script library includes one or more specific functions or algorithms, and the user can call these algorithms in the script instructions to process data or perform some kind of calculation.
[0102] The script library selected by the user is the target script library, and different scripting languages have corresponding script library reference methods. Similarly, different scripting languages also have corresponding execution strings used to call the algorithm in the script library; the algorithm called is the target algorithm. The following uses Python as an example to introduce a specific implementation method for step 203:
[0103] Assuming the target scripting language is Python, and the optional script libraries include a script library called remit_commom containing an algorithm called Util, then remit_commom is the target script library, and Util is the target algorithm. In the reference information, the target script library can be referenced using the import statement: "from remit_commom import Util".
[0104] Then you can call the Util algorithm to perform related actions in the Python script by using the string "Util".
[0105] 204. Generate corresponding input interfaces based on custom variables. The input interfaces are used to obtain the variable values of the custom variables.
[0106] It is understandable that step 204 is related to this application. Figure 1 The corresponding step 102 in the embodiment is described above and will not be repeated here.
[0107] To facilitate understanding, the above steps will be explained below with reference to the accompanying diagram.
[0108] Please see Figure 3 , Figure 3This is a schematic diagram of the action creation interface in the data construction method provided in the embodiments of this application.
[0109] In the action creation interface, users can fill in basic information about the action, including: a text box in the Project Name field where users can enter the project to which the action applies, helping users quickly identify the project to which the current action belongs when multiple projects exist; a drop-down menu in the Action Category field where users can select an action category that includes specifying the target scripting language, such as selecting Python scripting, which means the script action is written using Python-based scripting instructions; a text box in the Action Name field where users can enter the name of the action, helping users quickly identify specific actions when needed; and a text box in the Description field where users can enter a descriptive information about the action, such as an explanation of the script libraries, algorithms, data, etc. used in the script instructions. In team collaboration, a descriptive description can improve other users' understanding of the action.
[0110] Users can also fill in the script configuration information for the action on the creation interface. The script configuration information includes referenced script libraries, script commands, and a variable list. After determining the action category of the script action, the action creation interface includes optional script libraries corresponding to the current scripting language. Taking Python as an example, the optional script libraries section displays available script libraries through pip, the Python package management tool, allowing users to directly reference them and fill in the reference information in the text box corresponding to "Reference Script Library." The text box corresponding to the script command is used to input specific code. When a script library is referenced, the algorithm in the script library can be called through the corresponding string. In this embodiment, variables are defined using the format ${variable name}. Therefore, when the string ${variable name} is detected in the script command, it will be automatically identified and extracted, generating a corresponding input interface. This input interface is represented in the form of a variable list, which includes a serial number, variable name, variable value, and description. The input interface is specifically used to obtain the variable value.
[0111] In one possible implementation, prior to step 201, the following is also included:
[0112] In response to the installation operation of the target script library, the target script library is installed using a package management tool corresponding to the target script language;
[0113] Add the target script library as an optional script library.
[0114] Understandably, when it comes to scripting languages, various script libraries or packages are typically used to extend the functionality of the underlying language. These libraries or packages provide additional algorithms to implement specific functionalities. Each scripting language usually has one or more tools for installing and managing script libraries or packages, known as package managers. For example, Python's package manager is pip, used to install and manage Python packages, i.e., Python libraries; PHP's early package manager was PEAR, which has been gradually replaced by Composer; and Lua's package manager is LuaRocks, used to install and manage Lua libraries and programs.
[0115] Using Python as an example, suppose you want to install a script library called `requests` (the target script library), which is used to make HTTP requests in Python. Python's package manager is pip. You can install the `requests` script library using the following command: `pip install requests`.
[0116] This command tells the pip package manager to find and install the requests library.
[0117] In one possible implementation, prior to step 203, the following is also included:
[0118] In response to a version modification operation on the target script library, update the target script library to the specified version via the code repository.
[0119] It is understandable that there may be differences in functionality and algorithms between different versions of a script library. When referencing algorithms from a target script library, attention should be paid to whether the versions are compatible. Figure 3 As shown, the characters after "==" in the optional script library indicate the version of the script library. If the current version is not the version required by the user, the user can initiate a version modification operation, which is used to instruct the target script library to be updated to the specified version through the code repository.
[0120] A code repository is a central repository used to store and manage code and its dependencies, such as GitHub, GitLab, and Bitbucket. Script libraries are typically obtained from code repositories through package managers (such as npm, yarn, pip, composer, etc.). By using a package manager or pulling code directly from a code repository, you can update libraries or packages in your local project to specified versions.
[0121] Using Python scripts and pip as an example, modifying package versions in a code repository (such as GitHub) can trigger a pip update pipeline to perform a version update operation. See also... Figure 4 ,Figure 4 This is a flowchart of the pip update process in the data construction method provided in the embodiments of this application.
[0122] 401. Modify the version of the script library through the code repository.
[0123] User-defined Python script commands can be maintained in a Git repository, facilitating code review and pipeline integration. When a user needs to modify the version of the script library, they typically create a new branch (called a "personal branch" or "feature branch") from the main branch (or "main branch"). The version number of the script library is updated on this branch. After necessary testing and review, the personal branch is merged back into the main branch. This ensures that the main branch always contains the latest, tested code, while avoiding potential "pollution" or "chaos" that could result from direct modifications to the main branch.
[0124] 402. Initiate merge requests (MR).
[0125] Once a user has completed modifications to the script library version on their personal branch, and these modifications have been thoroughly tested and verified, they will typically initiate a MapReduce (MR) to merge these changes into the trunk branch (usually the master or main branch).
[0126] 403. Approved and incorporated into the main branch.
[0127] Once a merge request (MR) is created, it typically undergoes a review process to ensure the accuracy and reliability of the code changes. If the merge request passes the review and receives the necessary approvals, it can then be merged into the main branch.
[0128] 404. The merge into the main branch event was captured, triggering the pip package update pipeline to execute.
[0129] When a code repository management platform (such as GitLab, GitHub, Bitbucket, etc.) detects that code in a code repository has been merged into the main branch, it triggers a specific pipeline to execute a series of tasks (or stages). These tasks define the steps required to update and release the script library, so that users can install this version of the script library via pip.
[0130] 405. Compare the current script library version with the script library version in the repository trunk;
[0131] 406. If there are differences, update the script library version.
[0132] After the updated script library code is merged into the main branch event, version difference comparison can be performed. If differences exist, the current script library version will be updated to the version of the script library in the main branch of the repository, and pip management will be provided.
[0133] 407. Show users the updated script library.
[0134] Once the version update in pip is complete, the updated version will be displayed in the optional script library and made visible to the user.
[0135] The above describes an implementation of a script library version update method provided in this application. The following section will continue to introduce the data construction method provided in this application. Please refer to the documentation for further details. Figure 2 .
[0136] 205. Create script actions based on script instructions.
[0137] It is understandable that step 205 is related to this application. Figure 1 The corresponding step 103 in the embodiment is described above and will not be repeated here. Please refer to... Figure 3 In the action creation interface, when the user clicks the "Save" button, the script instruction is created as a script action.
[0138] 206. In response to the rule creation operation of the data construction rule, display the rule creation interface. The data construction rule includes at least one step of execution action. The script action corresponds to the target step in the data construction rule. The rule creation interface includes a step addition window.
[0139] 207. Add the script action to the step addition window corresponding to the target step.
[0140] It is understood that steps 206 and 207 are related to this application. Figure 1 This corresponds to step 105 in the corresponding embodiment.
[0141] In this embodiment, on the data construction platform, users can also create data construction rules as needed. For example, they can initiate a rule creation operation for a data construction rule, which can be triggered by a button, menu item, or other user interface element. Since data construction rules typically include one or more execution actions, when a user clicks or selects a rule creation operation, the data construction platform will display a rule creation interface. This interface includes a step addition window, allowing users to add execution actions to each step. The step corresponding to the script action is the target step, meaning the script action is added in the step addition window corresponding to the target step. After adding the execution actions for each step, the rule can be saved as the corresponding data construction rule.
[0142] For easier understanding, please refer to Figure 5 , Figure 5 This is a schematic diagram of the rule creation interface in the data construction method provided in the embodiments of this application.
[0143] In the rule creation interface, users can fill in the basic information for the data construction rule. Specifically: the "Project Name" field has a text box where users can enter the project to which the rule applies; this helps users quickly identify the rule when multiple projects exist. The "Rule ID" field has a text box where users can enter the rule's unique identifier, allowing them to locate the rule by ID. The "Rule Name" field has a text box where users can enter the name of the rule, helping them quickly identify specific actions when needed. The "Description" field has a text box where users can enter a description of the rule, such as explanations of the action types, algorithms, and data used. In team collaboration, this description can improve other users' understanding of the rule.
[0144] The rule creation interface includes a step addition window, specifically a "Add Step" button. Clicking this button displays saved actions, including the script actions generated in the previous steps, allowing users to add these script actions to the corresponding steps. After adding the action, a series of parameters and included custom variables are displayed. Users can modify the action using the "Modify Step Information" button. When modifying a script action, users can directly jump to... Figure 3 The corresponding interface.
[0145] After the data construction rule is created, the user can click the "Debug" button to execute the rule, or click the "Save" button to save the rule. Before "Debug," the user can select a debugging environment, including a development environment and a testing environment. Debugging requests in different environments will control the environment variables of the Python executor for the current task, thus achieving environment independence.
[0146] 208. When executing the target step of the data construction rule, the target algorithm is called to execute the script action based on the variable values.
[0147] It is understandable that step 208 is related to this application. Figure 1 This corresponds to step 106 in the corresponding embodiment. Since the script instructions include an execution string corresponding to the target algorithm in the target script library, the target algorithm needs to be called to execute the script action.
[0148] In one possible implementation, prior to step 208, the following is also included:
[0149] In response to the execution of the data construction rules, return the input interface;
[0150] Obtain variable values through the input interface.
[0151] Understandably, since variable values can be customized according to user needs, the input interface can be provided to the user both during script action creation and before executing data construction rules, improving the accuracy of rule execution. This input interface can be in the form of a pop-up window, appearing in response to the user clicking the "Debug" button. The user can then enter or check the values of custom variables through the editable text boxes in the pop-up window.
[0152] 209. Generate and execute the runtime log corresponding to the data construction rules. The runtime log includes script instructions.
[0153] 210. Output the runtime log.
[0154] Understandably, considering that users construct data using programmatic scripts, they may need to debug these scripts. Therefore, an execution log for the data construction rule can be generated and saved to a specific storage area to facilitate subsequent backtracking by users. This avoids blocking users' self-troubleshooting path due to the lack of complete data construction task execution logs. There is a one-to-one mapping relationship between data construction tasks and logs, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of the running log file in the data construction method provided in the embodiments of this application.
[0155] For ease of understanding, the following uses a Python script as an example to illustrate a specific application of the data construction method provided in an embodiment of this application. This application corresponds to a specific data construction architecture. Please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of the data construction architecture provided in the embodiments of this application.
[0156] This data architecture includes an application programming interface (API) gateway between clients and programs, rule receivers, rule processors, rule management, and a Python execution environment. Specifically:
[0157] The client is the data construction platform, which can be accessed through software development kits (SDKs) and web pages. Web pages are mainly used for the maintenance, management, and debugging of data construction rules, while SDKs are mainly used to help developers (users) better embed data construction capabilities into test case code.
[0158] API gateways are used to distribute access through load balancing, making full use of backend server resources and avoiding service unavailability caused by uneven traffic flow.
[0159] The rule receiver includes two functions: parsing rules and executing rules. The rule parsing function is used to read the rule's metadata from the database based on the rule ID and assemble the packet data according to the rule processor interface protocol. The rule execution function is used to call the rule processor to execute the rule interface.
[0160] The rule processor consists of two functions: worker processes and execution rules. The worker process is used to handle requests in a multi-process mode, aiming to improve service throughput and concurrency. The execution rules take into account that user-defined Python scripts are uncontrollable and may pose a risk of static resource leakage. If the worker process main process is reused, the database connection may not be released after the task is completed, which may cause the database to crash. Therefore, the execution rules can be carried out by an independent child process created (forked) by the worker process main process.
[0161] Rule management is used to perform CRUD operations on rules, steps, and actions, and this data is relied upon by the rule receiver.
[0162] The Python execution environment includes the installation of pip packages (script libraries) and hot updates of pip packages, ensuring the execution of each algorithm in the script instructions.
[0163] Please see Figure 8 , Figure 8 The execution flowchart of the data construction rules provided in the embodiments of this application includes:
[0164] 1. The SDK calls the rule execution interface.
[0165] To make the SDK easy to use, when calling the API Gateway rule execution interface, only the corresponding rule ID needs to be transmitted as a parameter.
[0166] 2. The rule receiver obtains rule metadata.
[0167] The rule metadata corresponding to the rule ID passed through by the SDK is read from the database. This includes the step data contained in the rule, the action data mapped to the step, and the Python script instructions to be executed. Key database tables can be listed as follows: Figure 9 As shown, Figure 9 This is a schematic diagram of the database table design in the data construction rules provided in the embodiments of this application.
[0168] 3. The rule receiver calls the execution rule RPC interface.
[0169] After obtaining the corresponding rule metadata in step 2, the request data is assembled according to the rule executor's interface protocol to call the RPC interface.
[0170] 4. The rule executor performs parameter validation.
[0171] This includes adding non-empty checks for required fields, validating the values of enumerated fields, blocking malicious and invalid requests, and preventing normal traffic from being squeezed out due to the occupation of service resources.
[0172] 5. The rule executor performs variable parsing.
[0173] To ensure the flexibility and versatility of the rules, rule execution needs to support external parameter injection, while providing commonly used built-in functions to meet users' needs for dynamically generated class parameters. Frequently used built-in functions include: getting the current system timestamp, encryption, signing, decryption, string splitting, etc.
[0174] 6. The rule executor executes user-defined Python scripts.
[0175] Execute user-defined Python scripts. Since the scripts executed for each task are dynamic, the built-in Python function `exec` is used to run the dynamic scripts. If a rule has multiple steps, all steps are executed sequentially using a recursive approach.
[0176] 7. The rule executor uploads logs.
[0177] Considering that users construct data using programmatic Python scripts and have a need to debug scripts, logs are uploaded and collected to facilitate backtracking for users. This avoids blocking users' self-troubleshooting path due to incomplete task execution logs. There is a one-to-one mapping relationship between tasks and logs.
[0178] 8. The rule executor assembly result is returned.
[0179] Final result: The steps in a single rule are executed sequentially, and the result of the last step is the content that the user cares about most.
[0180] Global variables: Considering the flexibility and complexity of user scenarios, data output to global variables during the execution of steps is also given to the user as response data;
[0181] Log information: Users can use this information to trace the execution process of the task, making it easier to troubleshoot and locate problems.
[0182] The data construction apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 10 . Figure 10 This is a schematic diagram of the data construction apparatus 1000 provided in the embodiments of this application. The data construction apparatus 1000 includes:
[0183] The acquisition module 1001 is used to acquire script instructions based on the target scripting language. The script instructions include custom variables represented by preset characters.
[0184] The generation module 1002 is used to generate corresponding input interfaces based on custom variables. The input interfaces are used to obtain the variable values of the custom variables.
[0185] Create module 1003, which is used to create script actions based on script instructions;
[0186] The acquisition module 1001 is also used to add script actions to data construction rules. The data construction rules include execution actions for at least one step, and the script actions correspond to the target steps in the data construction rules.
[0187] Execution module 1004 is used to execute script actions based on variable values when executing the target steps of the data construction rules.
[0188] The data construction apparatus provided in this application, after acquiring script instructions, creates corresponding input interfaces based on the custom variables represented by preset characters, since the script instructions include custom variables, so that users can input variable values. Based on these variable values, script actions are dynamically executed, solving the complexity of configuration management and maintenance caused by different variable values in development and testing environments. This allows the constructed data construction rules to be applied to both development and testing environments, achieving environment independence and improving the versatility of data construction. Furthermore, script actions are created based on the script instructions. These script actions can be added to the data construction rules as a step within the rules. Both the script actions and the data construction rules can be saved and reused, reducing the need to repeatedly write the same logic and improving the flexibility of data construction. By decomposing the data processing flow into a series of steps and actions, the data processing flow can be more easily managed and adjusted, thereby meeting different data construction needs and achieving reusable and maintainable construction capabilities (script actions). This solves the problem that existing data construction platforms struggle to implement data construction in complex business scenarios.
[0189] In one possible implementation,
[0190] The acquisition module 1001 is specifically used to respond to the action creation operation, display the action creation interface, which includes a script editing window; and to acquire script instructions through the script editing window.
[0191] In this embodiment, a user can initiate an operation to create a new data structure execution action. This operation can be triggered by a button, menu item, or other user interface element. When the user clicks or selects the action creation operation, an action creation interface is displayed, which allows the user to define a new execution action, specifically a script action created based on script instructions.
[0192] In one possible implementation, the action creation interface also includes an optional script library;
[0193] The device also includes:
[0194] The acquisition module 1001 is also used to acquire reference information of the target script library. The target script library is an optional script library. The optional script library includes the target algorithm, and the script instructions include the execution string that calls the target algorithm.
[0195] The execution module 1004 is specifically used to call the target algorithm to execute script actions based on variable values when executing the target steps of the data construction rules.
[0196] In this embodiment of the application, the action creation interface also includes an optional script library, which may include one or more script libraries for the user to select. Each script library includes one or more specific functions or algorithms, and the user can call these algorithms in the script instructions to process data or perform some kind of calculation.
[0197] In one possible implementation, the device further includes:
[0198] The installation module is used to respond to the installation operation of the target script library by installing the target script library through the package management tool corresponding to the target script language; and by adding the target script library as an optional script library.
[0199] In this application embodiment, when scripting languages are involved, various script libraries or packages are typically used to extend the functionality of the base language. These script libraries or packages provide additional algorithms to implement certain functions, and these script libraries or packages can be installed through package management tools.
[0200] In one possible implementation, the device further includes:
[0201] The update module is used to respond to version modification operations on the target script library by updating the target script library to the specified version via the code repository.
[0202] In this embodiment of the application, there may be differences in the functions and algorithms between different versions of the script library. If the current version is not the version required by the user, the user can initiate a version modification operation, which is used to instruct the target script library to be updated to the specified version through the code repository.
[0203] In one possible implementation,
[0204] The acquisition module 1001 is specifically used to respond to the rule creation operation of data construction rules, display the rule creation interface, which includes a step addition window; and add script actions to the step addition window corresponding to the target step.
[0205] In this embodiment, users can also create data construction rules as needed, such as initiating a rule creation operation for a data construction rule. This operation can be triggered by a button, menu item, or other user interface element. Since data construction rules typically include one or more execution actions, when a user clicks or selects the rule creation operation, the data construction platform displays a rule creation interface. This interface includes a step addition window, allowing users to add execution actions to each step. The step corresponding to the script action is the target step; that is, the script action is added in the step addition window corresponding to the target step. After adding the execution actions for each step, the rule can be saved as the corresponding data construction rule.
[0206] In one possible implementation, the device further includes:
[0207] The log module is used to generate runtime logs corresponding to the executed data construction rules. The runtime logs include script instructions; the runtime logs are also output.
[0208] In this embodiment of the application, considering that the user uses programmatic script instructions to construct data, the user may have the need to debug the script instructions. Therefore, the running log of the data construction rule can be generated and saved to a specific storage area to facilitate the user's subsequent backtracking.
[0209] In one possible implementation, the device further includes:
[0210] The input module is used to respond to the execution of data construction rules and return an input interface; variable values can be obtained through the input interface.
[0211] In this embodiment of the application, since the variable values can be customized according to the user's needs, the input interface can be provided to the user either during the process of creating script actions or before executing the data construction rules, thereby improving the accuracy of executing the rules.
[0212] Figure 11 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.
[0213] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.
[0214] The steps performed by the server in the above embodiments can be based on this Figure 11 The server structure shown.
[0215] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0216] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0217] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0218] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0219] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0220] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0221] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data construction method, characterized in that, include: Obtain script instructions based on the target scripting language, wherein the script instructions include custom variables represented by preset characters; A corresponding input interface is generated based on the custom variable, and the input interface is used to obtain the variable value of the custom variable; Create script actions based on the script instructions; The script action is added to the data construction rule, which includes an execution action for at least one step, and the script action corresponds to the target step in the data construction rule. When executing the target step of the data construction rule, the script action is executed based on the variable value.
2. The method according to claim 1, characterized in that, The acquisition of script instructions based on the target scripting language includes: In response to an action creation operation, an action creation interface is displayed, which includes a script editing window; The script instructions are obtained through the script editing window.
3. The method according to claim 2, characterized in that, The action creation interface also includes an optional script library; After displaying the action creation interface in response to the action creation operation, the following is also included: Obtain reference information for the target script library, which belongs to the optional script library, which includes the target algorithm, and the script instructions include an execution string that calls the target algorithm; When executing the target step of the data construction rule, the script action is executed based on the variable value, including: When executing the target step of the data construction rule, the target algorithm is invoked to execute the script action based on the variable value.
4. The method according to claim 3, characterized in that, Before displaying the action creation interface in response to the action creation request, the following is also included: In response to the installation operation of the target script library, the target script library is installed using a package management tool corresponding to the target script language; Add the target script library as an optional script library.
5. The method according to claim 3, characterized in that, Before obtaining the reference information to the target script library, which belongs to the optional script library, the method further includes: In response to a version modification operation on the target script library, the target script library is updated to the specified version via the code repository.
6. The method according to claim 1, characterized in that, Adding the script action to the data construction rule includes: In response to the rule creation operation of the data construction rules, a rule creation interface is displayed, which includes a step addition window; Add the script action to the step addition window corresponding to the target step.
7. The method according to claim 1, characterized in that, After executing the script action based on the variable value during the target step of executing the data construction rule, the method further includes: Generate and execute an operation log corresponding to the data construction rules, the operation log including the script instructions; Output the aforementioned runtime log.
8. The method according to claim 1, characterized in that, Before executing the script action based on the variable value during the target step of executing the data construction rule, the method further includes: In response to the execution of the data construction rules, the input interface is returned; The variable value is obtained through the input interface.
9. A data construction apparatus, characterized in that, include: The acquisition module is used to acquire script instructions based on the target scripting language, wherein the script instructions include custom variables represented by preset characters; A generation module is used to generate a corresponding input interface based on the custom variable, wherein the input interface is used to obtain the variable value of the custom variable; A module is created to generate script actions based on the script instructions. The acquisition module is further configured to add the script action to the data construction rule, the data construction rule including at least one step of execution action, and the script action corresponding to the target step in the data construction rule; An execution module is used to execute the script action based on the variable value when executing the target step of the data construction rule.
10. A computer device, characterized in that, include: Memory and processor; The memory stores instructions that, when executed on the processor, implement the data construction method as described in any one of claims 1 to 8.
11. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the data construction method as claimed in any one of claims 1 to 8.
12. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the data construction method as described in any one of claims 1 to 8.