Data processing method and device and data query method and device

Abstract

The invention provides a data processing method and device, and relates to the technical field of big data. According to the specific implementation scheme, a configuration file of a rule decision project in a strategy management platform is obtained; determining cluster connection information of the target database cluster based on the strategy management platform; based on the configuration file, determining a function source code of a user-defined function of the rule decision item in the target database cluster; based on the cluster connection information and the function source code, the user-defined function is loaded into the target database cluster, and the rule processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data processing method and device, a data query method and device, an electronic device, and a computer-readable medium. Background Art

[0002] In recent years, with the rapid development of IT (Information Technology) in my country, the informatization processes of domestic enterprises have continued to improve. Business decisions in business operations are often composed of multiple business rules. How to flexibly and conveniently configure rules to manage business decisions is crucial in informatization management. Summary of the Invention

[0003] Provided are a data processing method and device, a data query method and device, an electronic device, and a computer-readable storage medium.

[0004] According to a first aspect, a data processing method is provided, which includes: obtaining a configuration file of a rule decision project in a policy management platform; determining cluster connection information of a target database cluster based on the policy management platform; determining function source code of a user-defined function of the rule decision project in the target database cluster based on the configuration file; and loading the user-defined function into the target database cluster based on the cluster connection information and the function source code.

[0005] According to a second aspect, a data query method is provided, which includes: obtaining data to be queried; detecting whether the data to be queried is related to a user-defined function; in response to detecting that the data to be queried is related to the user-defined function, sending the data to be queried to the user-defined function; and receiving a query result for the data to be queried fed back by the user-defined function.

[0006] According to a third aspect, a data processing device is provided, which includes: a file acquisition unit, configured to acquire a configuration file of a rule decision project in a policy management platform; a connection determination unit, configured to determine cluster connection information of a target database cluster based on the policy management platform; a code determination unit, configured to determine function source code of a user-defined function of the rule decision project in the target database cluster based on the configuration file; and a loading unit, configured to load the user-defined function into the target database cluster based on the cluster connection information and the function source code.

[0007] According to a fourth aspect, a data query device is provided, which includes: a data acquisition unit, configured to acquire data to be queried; a detection unit, configured to detect whether the data to be queried is related to a user-defined function; a sending unit, configured to send the data to be queried to the user-defined function in response to detecting that the data to be queried is related to the user-defined function; and a receiving unit, configured to receive a query result of the data to be queried fed back by the user-defined function.

[0008] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation of the first aspect or the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0010] The data processing method and apparatus provided by the embodiments of the present disclosure first obtain the configuration file of the rule decision project in the policy management platform; secondly, based on the policy management platform, determine the cluster connection information of the target database cluster; thirdly, based on the configuration file, determine the function source code of the user-defined function of the rule decision project in the target database cluster; finally, based on the cluster connection information and the function source code, load the user-defined function into the target database cluster. Thus, based on the configuration file of the rule decision project, determine the function source code of the user-defined function of the target database cluster; based on the policy management platform, determine the cluster connection information of the target database cluster, and based on the function source code and the cluster connection information, load the user-defined function into the target database cluster, thereby realizing a pluggable, templated user-defined function publishing method, facilitating data processing during the execution of the rule decision project and improving the data processing efficiency of the rule decision project.

[0011] The data query method and device provided by the embodiments of the present disclosure first obtain the data to be queried; secondly, detect whether the data to be queried is related to the user-defined function; thirdly, in response to detecting that the data to be queried is related to the user-defined function, send the data to be queried to the user-defined function; finally, receive the query result of the data to be queried fed back by the user-defined function, judge whether the data to be queried is related to the user-defined function, and obtain the query result through the user-defined function when the data to be queried is related to the user-defined function, thereby improving the query efficiency of the data to be queried.

[0012] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a flow chart of an embodiment of a data processing method according to the present disclosure;

[0015] Figure 2 is a schematic diagram of the disclosed policy management platform, target database cluster, and the data processing structure between the two;

[0016] Figure 3 It is a structural diagram of the translation of the disclosed configuration file;

[0017] Figure 4 is a flow chart of an embodiment of a data query method according to the present disclosure;

[0018] Figure 5 It is a structural diagram of the data query process of the present disclosure;

[0019] Figure 6 is a schematic structural diagram of an embodiment of a data processing device according to the present disclosure;

[0020] Figure 7 is a structural diagram of an embodiment of a data query device according to the present disclosure;

[0021] Figure 8 It is a block diagram of an electronic device used to implement the data processing method or data query method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0024] Modern information-based enterprises build business scenario strategy management platforms to uniformly manage the configuration, release, and execution of rule-based decision models in different business scenarios.

[0025] In an actual policy management platform, a rule decision model usually consists of multiple rules and a workflow that defines the order in which the rules are executed.

[0026] Technically, business scenario policy management platforms are typically implemented based on open-source workflow engine components (such as jBPM, Activiti, Camunda, and Flowable) and open-source rule expression engine components (such as Drools, Aviator, Mvel, and EasyRules). The rule expression engine is responsible for defining rules and storing them in a pseudocode configuration file with a specified syntax format. The workflow engine is responsible for defining workflows—the execution order of multiple rules—and storing them in an XML (Extensible Markup Language) configuration file based on the BPMN (Business Process Model and Notation) standard. Before the rule-based decision model is executed, the rule and workflow configuration files must be read and parsed. To improve the efficiency of processing large amounts of data, the execution of the rule-based decision model is often encapsulated as an operator within the big data computing engine.

[0027] In terms of the process, business personnel first configure the rules and workflows in the rule decision model on the front-end page of the business scenario policy management platform. The back-end service converts the rules and workflows into configuration files when the model is saved, and compiles and packages them into executable JAR (Java Archive, JAVA archive file) packages under the big data platform architecture when the model is released. Finally, developers use software programs and scripts to re-encapsulate and call the released JAR packages to implement business scenario decisions.

[0028] Low computational efficiency: Mainstream open-source rule expression engines and workflow engines use configuration files to store rules and workflows. During execution, in addition to the rule logic itself, they must first load and parse the rule and workflow configuration files. Then, they must use Java reflection, state control, and node transitions to drive the sequential execution of the rules. This additional computational cost leads to low rule execution efficiency.

[0029] The application threshold for rule-based decision models is high: The rule-based decision model is published as an executable JAR package. The final model needs to be repackaged using software programs and scripts before it can be used, which requires a high development threshold. The business scenario policy management platform itself is designed for self-configuration and use by business personnel and data analysts, who generally lack software development capabilities.

[0030] Poor scalability of support for big data computing engines: The business scenario policy management platform has a built-in supported big data computing engine.

[0031] In view of the above-mentioned defects of traditional technologies, the present disclosure provides a data processing method, which improves the efficiency of rule execution. Figure 1 A process 100 according to an embodiment of a data processing method of the present disclosure is shown. The data processing method includes the following steps:

[0032] Step 101: Obtain a configuration file of a rule decision project in a policy management platform.

[0033] In this embodiment, the policy management platform is a platform that can configure rules automatically or manually. The policy management platform can be used to configure rules for different rule decision projects. A rule decision project is a project that requires the execution of at least one rule. The rules in the rule decision project can be logical and sequential. For example, the rule decision project is project A. In project A, rule A1 must be executed first, and then rule A2. When rule A1 and rule A2 are executed in sequence, project A is completed.

[0034] In this embodiment, the rule decision project is a project corresponding to at least one rule. The project is similar to a model and has a corresponding main structure and code execution order.

[0035] In this embodiment, a configuration file is a file used to store and manage settings and parameters for software, systems, or applications. It typically contains a set of key-value pairs that specify various options, variables, and functions. Configuration files help developers more easily manage and maintain software, while also allowing users to customize the behavior of applications according to their needs. Configuration files are primarily used to store software parameters and settings, which determine the software's behavior and appearance. By modifying the configuration file, users can customize the software's functionality, such as defining constants, domain names, directory paths, and other information.

[0036] In this embodiment, in order to better manage rule decision projects, a workflow engine component and a rule expression engine component are provided in the policy management platform. The rule expression engine component can be used to define the rules of the rule decision project and store them in a pseudocode configuration file in a specified syntax format. The workflow engine component can be used to define the workflow of the rule decision project. The workflow is the execution order of at least one rule in the rule decision project, and the workflow is stored in a standard XML configuration file.

[0037] In this embodiment, the configuration file of the rule decision project is a configuration file related to the rule decision project, and is a file that stores the settings and parameters of the rule decision project. The settings and parameters of the rule decision project can be determined through the configuration file of the rule decision project. When the rule decision project includes only one rule, the configuration file of the rule decision project is a pseudocode configuration file that defines the rule; when the rule decision project includes multiple rules, the configuration file of the rule decision project includes a pseudocode configuration file and an XML configuration file of the workflow.

[0038] Step 102: Determine cluster connection information of the target database cluster based on the policy management platform.

[0039] In this embodiment, the policy management platform can have multiple databases, and the target database cluster is the database cluster associated with the rule-based decision project. Cluster connection information is key information for connecting to the target database cluster. This information facilitates connecting specific information to the target database cluster, thereby facilitating access to the target database cluster. A database cluster utilizes at least two or more database servers to form a virtual single database logical image. This virtual single database system provides transparent data services to clients, similar to a single database system, achieving the desired effect of a target database cluster.

[0040] In this embodiment, cluster connection information may include: connection strings, driver programs, load balancing and failover information, node management information, and data synchronization information. The connection string is key to connecting to a cluster database and typically contains the addresses of multiple database nodes to ensure automatic failover to other nodes if a node fails. For example, in a MySQL cluster, the connection string may contain multiple host names, port numbers, and other parameters. The driver program is also key to connecting to a cluster database. Different database management systems (DBMSs) provide different driver programs, and these drivers also vary in their support for cluster connections. Load balancing and failover information are core features of cluster databases. Load balancing intelligently distributes user requests to various database nodes, ensuring a relatively balanced load on each node, thereby improving overall system performance. The failover mechanism automatically transfers requests to other nodes when a node fails, ensuring high system availability. Node management information includes node addition, deletion, monitoring, and maintenance. By real-time monitoring of node metrics such as CPU usage, memory usage, and number of connections, abnormalities can be promptly detected and addressed to prevent system performance degradation. Data synchronization information is crucial for ensuring database cluster consistency. By synchronizing data changes on one node to other nodes in real time, data consistency across all nodes is ensured. Common data synchronization methods include master-slave replication, dual-master replication, and multi-master replication.

[0041] In this embodiment, the above step 102 includes: determining the release information of the policy management platform based on the policy management platform; determining, based on the release information, the target database cluster selected when the policy management platform is released, obtaining cluster connection information from the cluster management module to establish a connection with the target database cluster cluster through the cluster connection information, wherein the cluster management module is a module in the policy management platform that manages the database cluster.

[0042] Step 103: Determine the function source code of the user-defined function of the rule decision item in the target database cluster based on the configuration file.

[0043] In this embodiment, the target database cluster supports user-defined functions. User-defined functions allow users to create customized functions in the target database cluster. Loading the function source code of the user-defined function into the target database cluster can expand the functionality of the target database cluster, thereby providing more flexible data processing capabilities.

[0044] In this embodiment, the above step 103 includes: determining each rule in the rule decision project and the logical relationship between any two rules based on the configuration items of the configuration file; and compiling all rules and logical relationships in the rule decision project into function source code that is compatible with the user-defined function of the target database cluster.

[0045] Step 104 : Load the user-defined function into the target database cluster based on the cluster connection information and the function source code.

[0046] In this embodiment, the above-mentioned step 104 includes: compiling and packaging the function source code, that is, uniformly compiling the function source code generated in step 103 and packaging it into a function compression package; establishing a connection with the target database cluster according to the cluster connection information; uploading the generated function compression package to the target database cluster, and finally controlling the target database cluster to create a user-defined function, that is, loading the user-defined function into the target database cluster.

[0047] In this embodiment, user-defined functions are typically written in a programming language based on specific requirements and functionality. Once written, they can be added to a database or system using a specific method. Once added, other developers or applications can call these user-defined functions to perform specific tasks. This makes development more efficient because existing code can be reused, eliminating the need to rewrite the same logic each time.

[0048] The data processing method provided by the embodiments of the present disclosure first obtains the configuration file of the rule decision project in the policy management platform; secondly, based on the policy management platform, determines the cluster connection information of the target database cluster; thirdly, based on the configuration file, determines the function source code of the user-defined function of the rule decision project in the target database cluster; finally, based on the cluster connection information and the function source code, loads the user-defined function into the target database cluster. Thus, based on the configuration file of the rule decision project, determines the function source code of the user-defined function of the target database cluster; based on the policy management platform, determines the cluster connection information of the target database cluster, and based on the function source code and the cluster connection information, loads the user-defined function into the target database cluster, thus realizing a pluggable, templated user-defined function publishing method, facilitating data processing during the execution of the rule decision project and improving the data processing efficiency of the rule decision project.

[0049] In some optional implementations of the present disclosure, the above-mentioned determining the cluster connection information of the target database cluster based on the policy management platform includes: determining the database information of the target database cluster of the rule decision project based on the policy management platform; and determining the cluster connection information of the target database cluster based on the database information.

[0050] In this optional implementation, database information refers to all data and related metadata stored in the target database cluster. Database information includes table structures, field definitions, data records, indexes, constraints, views, stored procedures, triggers, and so on. Data in the target database cluster can be accessed from any server in the cluster. Data can be distributed across different nodes to improve data reliability and availability.

[0051] In this optional implementation, determining the cluster connection information of the target database cluster based on the database information includes: logging into the target database cluster based on the database information; sending an access request for obtaining the cluster connection information to the target database cluster, and receiving the cluster connection information of the target database cluster.

[0052] The method for determining cluster connection information provided by this optional implementation method determines the database information of the target database cluster of the rule decision project based on the policy management platform; determines the cluster connection information of the target database cluster based on the database information, determines the database information from the policy management platform, and determines the cluster connection information based on the database information, thereby improving the reliability of obtaining the cluster connection information.

[0053] In some optional implementations of this embodiment, the above-mentioned determination of the function source code of the user-defined function of the rule decision project in the target database cluster based on the configuration file includes: translating the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code; and determining the function source code of the user-defined function of the programming language in the target database cluster based on the programming language source code.

[0054] In this optional implementation, the language format of the programming language source code can be determined by the language supported by the format of the configuration file. For example, the rule expression engine component is Drools, and its rule configuration file is in DRL format. The pseudocode syntax in DRL supports both JAVA and MVEL. Therefore, the rule configuration file can be translated into JAVA or MVEL source code.

[0055] like Figure 2 As shown, the processing structure located between the policy management platform and the target database cluster includes a model translation module, a UDF (User-Defined Function) generation module, and a model push module, wherein the data processing structure is a device corresponding to the data processing method disclosed in the present invention, and the model translation module is responsible for translating the configuration file of the rule decision project into high-level programming language source code.

[0056] The method for determining function source code provided by the embodiment of the present disclosure translates the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code; based on the programming language source code, the function source code of the user-defined function of the programming language in the target database cluster is determined. To this end, the configuration file is translated into programming language source code, and then the function source code is determined through the programming language source code, thereby improving the reliability of obtaining the function source code.

[0057] Optionally, the determining, based on the configuration file, the function source code of the user-defined function of the rule decision item in the target database cluster includes: translating the configuration file into the function source code of the user-defined function of the target database cluster.

[0058] In some embodiments of the present disclosure, the above-mentioned translation of the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code includes: based on the configuration file, determining at least one rule of the rule decision project, the execution logic information of each rule in the at least one rule, the workflow of at least one rule, and the configuration location information of each rule in the workflow; based on at least one rule, constructing a rule class body code; based on the execution logic information, filling the first branch structure code corresponding to the execution logic information in the rule class body code to obtain the rule class structure code; based on the workflow, constructing the workflow class body code; based on the configuration location information, filling the second branch structure code corresponding to the configuration location information in the workflow class body code to obtain the workflow class structure code; using the rule class structure code and the workflow class structure code as programming language source code.

[0059] In this optional implementation, the execution logic information is the content information of the rule, and the configuration location information is information reflecting the order and nesting relationship of the rules in the workflow.

[0060] In this optional implementation, based on at least one rule, constructing the rule class body code includes the following steps 1.1.1 to 1.1.3:

[0061] Step 1.1.1: Define the rule class's namespace and class name, and maintain the mapping between the original DRL file and the rule class. To simplify the mapping, use the package statement in the DRL file as the namespace, completing the semicolon to ensure syntax compliance. Use the DRL file attribute "ruleflow-group" as the class name so that it can be called by the workflow. If multiple package statements exist, the rule class is generated for each namespace.

[0062] Step 1.1.2: Introduce dependencies and extract import statements in DRL.

[0063] Step 1.1.3: Define the function and replace the keyword function in the function module with public.

[0064] In this optional implementation, the above-mentioned filling of the first branch structure code corresponding to the execution logic information in the rule class body code based on the execution logic information refers to translating the rule logic into the execute method. DRL rules are composed of one or more minimization rules, each of which begins with rule and ends with end. The translation of the rules requires traversing all the minimization rules in the DRL file. The above-mentioned filling of the first branch structure code corresponding to the execution logic information in the rule class body code based on the execution logic information to obtain the rule class structure code includes the following steps 1.2.1 to 1.2.3:

[0065] Step 1.2.1: Filter, that is, remove the minimization rules whose attribute enabled is false.

[0066] Step 1.2.2: Sorting, that is, determining the execution order of the minimization rules, sorting them from large to small according to the value of the attribute salience. If the salience values are the same, then sort them according to the order of declaration.

[0067] Step 1.2.3: Grouping. Minimization rules with the same activation-group value belong to the same group. Only one minimization rule within a group can be executed. For example, in activation-group A, declare a Boolean variable isA with a default value of false. Add the logical statement isA! = true to the trigger condition of all minimization rules in group A, and add the assignment statement isA = true to the execution action to ensure that at most one rule in group A is triggered.

[0068] In this optional implementation, in an actual production example, the workflow engine component can be jBPM, whose rule configuration file is in the BPMN2 standard XML format, and its core logic is defined in <process>Within the tag, the specific steps for constructing the workflow class body code include steps 2.1.1 to 2.1.2:

[0069] Step 2.1.1: Define the namespace and class name of the workflow class, and maintain the mapping relationship between the original BPMN file and the workflow class. <process>The id attribute value of the tag is used as the class name, using <process>The tns:packageName attribute value within the tag is used as the namespace.

[0070] Step 2.1.2: Introduce dependencies and extract <extensionelements>Tags <tns:import>Tag, using its name attribute value.

[0071] In this optional implementation, the second branch structure code for filling the corresponding configuration location information in the workflow class body code based on the configuration location information includes: calling the execute method of each element in the workflow in the execution order. The second branch structure code for filling the corresponding configuration location information in the workflow class body code based on the configuration location information includes steps 2.2.1 to 2.2.5:

[0072] Step 2.2.1: Locate the start and end nodes: <startevent>and <endevent>The tag, whose id attribute value represents the start node and end node respectively, with the start node as the current node.

[0073] Step 2.2.2: Locate the next node: traverse all <sequenceflow>Tag, find out the sourceRef attribute value of the current node, and its targetRef attribute value is the next node.

[0074] Step 2.2.3: Find the tag whose id attribute value is the targetRef from step 2.2. If the tag type is businessRuleTask, it represents a rule. Map it to the corresponding rule class based on the g:ruleFlowGroup attribute and call its execute method. If the tag type is callActivity, it represents a sub-workflow. Map it to the corresponding sub-workflow based on the calledElement attribute and call its execute method.

[0075] Step 2.2.4: Set targetRef as the current node.

[0076] Step 2.2.5: Repeat steps 2.2.2 to 2.2.4 until the current node is the end node or the corresponding label cannot be found in step 2.2.2.

[0077] An optional implementation of the present disclosure provides a method for translating a configuration file into a programming language source code. Based on the configuration file, at least one rule of a rule decision project, execution logic information of each rule in the at least one rule, a workflow of at least one rule, and configuration location information of each rule in the workflow are determined; based on the at least one rule, a rule class body code is constructed; based on the execution logic information, a first branch structure code corresponding to the execution logic information is filled in the rule class body code to obtain a rule class structure code; based on the workflow, a workflow class body code is constructed; based on the configuration location information, a second branch structure code corresponding to the configuration location information is filled in the workflow class body code to obtain a workflow class structure code; and the rule class structure code and the workflow class structure code are used as programming language source code, thereby improving the reliability of obtaining the programming language source code.

[0078] Optionally, the above-mentioned translation of the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code includes: based on the configuration file, determining the rules of the rule decision project and the execution logic information of the rules; based on at least one rule, constructing the rule class main body code; based on the execution logic information, filling the first branch structure code corresponding to the execution logic information in the rule class main body code to obtain the rule class structure code; and using the rule class structure code as the programming language source code.

[0079] In some optional implementations of the present disclosure, the above-mentioned translation of the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code also includes: determining the main workflow class structure code in the workflow class structure code; generating the rule decision project class code; and starting the starting logic in the rule decision project class code with the main workflow class structure code.

[0080] In this optional implementation, in order to improve the standardization and readability of the model translated code, three parent classes are defined: Rule, Flow, and Model. After translation into a high-level programming language, the relationship between rules, workflows, and models is as follows: Figure 3 As shown in the figure, Rule represents the rule class, Flow represents the workflow class, and Model represents the rule decision project class. The rule class source code (including the rule class structure code) inherits from Rule. The variable ruleName declares the rule name, and the method execute (such as Figure 3 The execute(Record) in the example is composed of one or more IF branch structures that represent minimization rules. The conditional expression in a single branch structure represents the triggering condition of the minimization rule, and the statements in the branch structure represent the execution action of the minimization rule. The source code of the workflow class (including the workflow class structure code) is inherited from Flow. The variable flowName declares the workflow name. The execute method (such as Figure 3 In execute(Record)), the execute method of the rule and sub-workflow is called in sequence according to the configuration order. The model class source code (including the rule decision project class code) inherits from Model. The variable modelName declares the model or business scenario name, the variable startPoint declares the starting workflow, and the method start (such as Figure 3 In start(Record)), the execute method of startPoint is called.

[0081] In this optional implementation, a model class can be generated by the model translation module, where the model class is the class corresponding to the rule decision project class code. Specifically, the above-mentioned starting logic in the rule decision project class code starts with the main workflow class structure code, including steps 3.1 and 3.2:

[0082] Step 3.1: Obtain the main workflow class. From the business scenario policy management platform, obtain the configured main workflow, and then find the main workflow class based on the mapping logic in step 2.1.1 above.

[0083] Step 3.2: Generate the model class. Assign the main workflow instance to the startPoint variable of the model class, and the start method of the model class calls the execute method of startPoint.

[0084] The optional implementation of the present disclosure provides a method for translating programming language source code, which determines the main workflow class structure code in the workflow class structure code; generates the rule decision project class code; and uses the main workflow class structure code as the starting logic in the rule decision project class code, providing a reliable implementation method for obtaining the programming source code.

[0085] In some optional implementations of the present disclosure, the above-mentioned determination of the function source code of the user-defined function of the programming language in the target database cluster based on the programming language source code includes: obtaining the database type of the user-defined function; based on the database type, determining the function source code template and the database schema definition language template; based on the programming language source code, replacing the parameters of the function source code template and the database schema definition language template to obtain the function source code of the user-defined function.

[0086] In this optional implementation, if Figure 2 As shown, the UDF generation module includes a UDF template management submodule and a UDF encapsulation submodule. The UDF encapsulation submodule performs UDF encapsulation. The UDF template management submodule replaces the reserved model name in the corresponding function source code template and database schema definition language template with the actual model class name based on the UDF database type selected when the model is published, and generates the function source code of the user-defined function.

[0087] In this optional implementation, the UDF template management submodule configures and manages function source code templates and database schema definition language templates to support hot-swappable expansion of multiple big data SQL engines such as Hive, Spark, and Flink. The UDF encapsulation submodule is responsible for replacing the reserved model name in the template with the actual model class name to generate the function source code of the user-defined function.

[0088] The optional implementation method of the present disclosure provides a method for obtaining function source code, which obtains the database type of the user-defined function; based on the database type, determines the function source code template and the database model definition language template; based on the programming language source code, replaces the parameters of the function source code template and the database model definition language template to obtain the function source code of the user-defined function, thereby improving the reliability of obtaining the function source code.

[0089] In some optional implementations of the present disclosure, the above-mentioned loading of the user-defined function into the target database cluster based on the cluster connection information and the function source code includes: packaging the function source code to obtain a user-defined function package; establishing a connection with a server cluster of the target database cluster based on the cluster connection information; and uploading the user-defined function package to the server cluster to create the user-defined function in the server cluster.

[0090] In this optional implementation, if Figure 2 As shown in the figure, the model push module consists of a cluster management submodule, a compilation and packaging submodule, and a model push submodule. The cluster management submodule is responsible for maintaining and managing cluster connection information for each database cluster; the compilation and packaging submodule is responsible for compiling the translated source code of the user-defined function and packaging it into a JAR file; the model push function is responsible for connecting to the target database cluster, uploading the JAR file to the target database cluster, and creating the user-defined function in the target database cluster using the database schema definition language template.

[0091] This optional implementation provides a method for loading user-defined functions, packaging function source code to obtain a user-defined function package; establishing a connection with a server cluster of a target database cluster based on cluster connection information; uploading the user-defined function package to the server cluster to create a user-defined function in the server cluster, thereby improving the reliability of obtaining the user-defined function.

[0092] Figure 4 A process 400 according to an embodiment of the data query method of the present disclosure is shown. The data processing method includes the following steps:

[0093] Step 401: Obtain the data to be queried.

[0094] In this embodiment, the data query method is applied to a target database cluster to realize the query of data in the target data cluster. The target database cluster is a database cluster related to the rule-based policy project. Each target database in the target database cluster supports at least one user-defined function. The user-defined function allows users to create customized functions in the target database cluster. The function source code of the user-defined function is loaded into the target database cluster to expand the function of the target database cluster, thereby providing more flexible data processing capabilities.

[0095] In this embodiment, the data to be queried is data of a database to be queried. By querying the target database cluster with the data to be queried, a query result of the target database can be obtained.

[0096] Step 402: Detect whether the data to be queried is related to the user-defined function.

[0097] In this embodiment, the data corresponding to the user-defined function has a corresponding format. By checking whether the format of the data to be queried is consistent with the format of the user-defined function, it is detected whether the data to be queried is related to the user-defined function.

[0098] In this embodiment, the above-mentioned step 402 includes: detecting whether the number of fields of the data to be queried is the same as the preset number of fields; if it is the same as the preset number of fields, detecting whether the data type of the field of the data to be queried is the same as the preset data type; if it is the same as the preset data type, detecting whether the format of the field of the data to be queried meets the format requirements of the user-defined function; if it meets the format requirements, determining that the data to be queried is related to the user-defined function.

[0099] Step 403: In response to detecting that the data to be queried is related to the user-defined function, the data to be queried is sent to the user-defined function.

[0100] In this embodiment, in response to detecting that the data to be queried is related to the user-defined function, the data to be queried is sent to the target database cluster, and then sent to the user-defined function therein through the target database cluster.

[0101] Step 404: Receive the query result of the query data fed back by the user-defined function.

[0102] In this embodiment, after the user-defined function obtains the data to be queried, it executes the corresponding query logic of the user-defined function itself in the target database cluster to obtain the query result, and feeds back the query result of the data to be queried to the execution entity on which the user-defined function runs.

[0103] From the perspective of usage process, the application of rule decision model is as follows Figure 5 First, the business personnel configure the rule decision project on the policy management platform, and publish the rule decision project after the test passes. When the rule decision project is published, it will call Figure 2 The model translation module, UDF encapsulation module, and model push module in the data processing structure shown push rule decision projects to the target database cluster as user-defined functions. The user-defined functions are then called within the SQL logic for data processing, and the query results are finally read. This enables self-service throughout the entire process of rule decision model configuration, release, and application.

[0104] The data query method provided by the embodiment of the present disclosure first obtains the data to be queried; secondly, detects whether the data to be queried is related to the user-defined function; thirdly, in response to detecting that the data to be queried is related to the user-defined function, sends the data to be queried to the user-defined function; finally, receives the query result of the data to be queried fed back by the user-defined function, judges whether the data to be queried is related to the user-defined function, and obtains the query result through the user-defined function when the data to be queried is related to the user-defined function, thereby improving the query efficiency of the data to be queried.

[0105] Further references Figure 6 As an implementation of the data processing method shown in the above figures, the present disclosure provides an embodiment of a data processing device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0106] like Figure 6 As shown, the data processing device 600 provided in this embodiment includes: a file acquisition unit 601, a connection determination unit 602, a code determination unit 603, and a loading unit 604. The file acquisition unit 601 can be configured to acquire a configuration file of a rule decision project in a policy management platform. The connection determination unit 602 can be configured to determine the cluster connection information of a target database cluster based on the policy management platform. The code determination unit 603 can be configured to determine the function source code of a user-defined function of a rule decision project in the target database cluster based on the configuration file. The loading unit 604 can be configured to load the user-defined function into the target database cluster based on the cluster connection information and the function source code.

[0107] In this embodiment, the specific processing of the file acquisition unit 601, the connection determination unit 602, the code determination unit 603, and the loading unit 604 and the technical effects thereof can be referred to in the respective Figure 1 The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0108] In some optional implementations of this embodiment, the connection determination unit 602 is configured to: determine database information of the target database cluster of the rule decision item based on the policy management platform; and determine cluster connection information of the target database cluster based on the database information.

[0109] In some optional implementations of this embodiment, the above-mentioned code determination unit 603 is configured to: translate the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code; based on the programming language source code, determine the function source code of the user-defined function of the programming language in the target database cluster.

[0110] In some optional implementations of this embodiment, the above-mentioned code determination unit 603 is configured to: determine at least one rule of the rule decision project, the execution logic information of each rule in the at least one rule, the workflow of at least one rule, and the configuration location information of each rule in the workflow based on the configuration file; construct a rule class body code based on at least one rule; based on the execution logic information, fill in the first branch structure code corresponding to the execution logic information in the rule class body code to obtain the rule class structure code; based on the workflow, construct a workflow class body code; based on the configuration location information, fill in the second branch structure code corresponding to the configuration location information in the workflow class body code to obtain the workflow class structure code; use the rule class structure code and the workflow class structure code as programming language source code.

[0111] In some optional implementations of this embodiment, the above-mentioned code determination unit 603 is configured to: determine the main workflow class structure code in the workflow class structure code; generate the rule decision project class code; and start the starting logic in the rule decision project class code with the main workflow class structure code.

[0112] In some optional implementations of this embodiment, the above-mentioned code determination unit 603 is configured to: obtain the database type of the user-defined function; determine the function source code template and the database schema definition language template based on the database type; replace the parameters of the function source code template and the database schema definition language template based on the programming language source code to obtain the function source code of the user-defined function.

[0113] In some optional implementations of this embodiment, the above-mentioned loading unit 604 is configured to: package the function source code to obtain a user-defined function package; establish a connection with the server cluster of the target database cluster based on the cluster connection information; upload the user-defined function package to the server cluster to create a user-defined function in the server cluster.

[0114] The data processing device provided by the embodiment of the present disclosure includes: first, a file acquisition unit 601 acquires a configuration file of a rule decision project in a policy management platform; second, a connection determination unit 602 determines the cluster connection information of a target database cluster based on the policy management platform; third, a code determination unit 603 determines the function source code of a user-defined function of the rule decision project in the target database cluster based on the configuration file; and finally, a loading unit 604 loads the user-defined function into the target database cluster based on the cluster connection information and the function source code. Thus, based on the configuration file of the rule decision project, the function source code of the user-defined function of the target database cluster is determined; based on the policy management platform, the cluster connection information of the target database cluster is determined, and based on the function source code and the cluster connection information, the user-defined function is loaded into the target database cluster, thereby realizing a pluggable and templated user-defined function publishing method, facilitating data processing during the execution of the rule decision project, and improving the data processing efficiency of the rule decision project.

[0115] Further references Figure 7 As an implementation of the data query method shown in the above figures, the present disclosure provides an embodiment of a data query device. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0116] like Figure 7 As shown, the data query device 700 provided in this embodiment includes: a data acquisition unit 701, a detection unit 702, a sending unit 703, and a receiving unit 704. The data acquisition unit 701 can be configured to acquire the data to be queried. The detection unit 702 can be configured to detect whether the data to be queried is relevant to a user-defined function. The sending unit 703 can be configured to, in response to detecting that the data to be queried is relevant to the user-defined function, send the data to be queried to the user-defined function. The receiving unit 704 can be configured to receive the query result of the data to be queried fed back by the user-defined function.

[0117] In this embodiment, the specific processing of the data acquisition unit 701, the detection unit 702, the sending unit 703, and the receiving unit 704 and the technical effects thereof can be referred to in the respective Figure 4 The relevant descriptions of step 401, step 402, step 403, and step 404 in the corresponding embodiment are not repeated here.

[0118] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0120] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0121] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0122] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a data processing method or a data query method. For example, in some embodiments, the data processing method or the data query method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method or the data query method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the data processing method or the data query method in any other appropriate manner (for example, by means of firmware).

[0124] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device or a data query device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0130] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0131] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.< / sequenceflow> < / endevent> < / startevent> < / tns:import> < / extensionelements> < / process> < / process> < / process>

Claims

1. A data processing method, comprising: Obtain the configuration file of the rule decision project in the policy management platform; Determining cluster connection information of a target database cluster based on the policy management platform; Determining, based on the configuration file, a function source code of a user-defined function of the rule decision item in the target database cluster; The user-defined function is loaded into the target database cluster based on the cluster connection information and the function source code.

2. The method according to claim 1, wherein Determining the cluster connection information of the target database cluster based on the policy management platform includes: Determining database information of a target database cluster of the rule decision project based on the policy management platform; Based on the database information, cluster connection information of the target database cluster is determined.

3. The method according to claim 1, wherein Determining, based on the configuration file, the function source code of the user-defined function of the target database cluster for the rule decision item includes: translating the configuration file into programming language source code to determine the relationship between the various rules in the rule decision project through the programming language source code; Based on the programming language source code, a function source code of a user-defined function of the programming language in the target database cluster is determined.

4. The method according to claim 3, wherein: The translating the configuration file into programming language source code to determine the relationship between the rules in the rule decision project through the programming language source code includes: Determining, based on the configuration file, at least one rule of the rule decision project, execution logic information of each rule in the at least one rule, a workflow of the at least one rule, and configuration location information of each rule in the workflow; Based on the at least one rule, construct a rule class body code; Based on the execution logic information, filling the first branch structure code corresponding to the execution logic information in the rule class main body code to obtain the rule class structure code; Based on the workflow, construct the workflow class main code; Based on the configuration location information, filling the second branch structure code corresponding to the configuration location information in the workflow class main body code to obtain the workflow class structure code; The rule-class structure code and the workflow-class structure code are used as the programming language source code.

5. The method according to claim 4, wherein The translating the configuration file into programming language source code to determine the relationship between the rules in the rule decision project through the programming language source code further includes: Determining a main workflow class structure code in the workflow class structure code; Generate rule decision project class code; The starting logic in the rule decision project class code starts with the main workflow class structure code.

6. The method according to claim 3, wherein: The determining, based on the programming language source code, the function source code of the user-defined function of the programming language in the target database cluster includes: Obtaining the database type of the user-defined function; Based on the database type, determining a function source code template and a database schema definition language template; Based on the programming language source code, the parameters of the function source code template and the database schema definition language template are replaced to obtain the function source code of the user-defined function.

7. The method according to any one of claims 1 to 6, wherein: The step of loading the user-defined function into the target database cluster based on the cluster connection information and the function source code includes: Packing the function source code to obtain a user-defined function package; Establishing a connection with a server cluster of the target database cluster based on the cluster connection information; The user-defined function package is uploaded to the server cluster to create the user-defined function in the server cluster.

8. A data query method, comprising: Get the data to be queried; Detecting whether the data to be queried is related to a user-defined function; In response to detecting that the data to be queried is related to a user-defined function, sending the data to be queried to the user-defined function; Receive a query result for the data to be queried fed back by the user-defined function.

9. A data processing device, comprising: A file acquisition unit configured to acquire a configuration file of a rule decision project in the policy management platform; a connection determination unit configured to determine cluster connection information of a target database cluster based on the policy management platform; A code determination unit is configured to determine, based on the configuration file, a function source code of a user-defined function of the rule decision item in the target database cluster; A loading unit is configured to load the user-defined function into the target database cluster based on the cluster connection information and the function source code.

10. A data query device, comprising: A data acquisition unit is configured to acquire data to be queried; A detection unit configured to detect whether the data to be queried is related to a user-defined function; a sending unit, configured to, in response to detecting that the data to be queried is related to a user-defined function, send the data to be queried to the user-defined function; The receiving unit is configured to receive a query result of the to-be-queried data fed back by the user-defined function.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

Images