A massive data decision engine system and method based on Java

CN118779115BActive Publication Date: 2024-11-05紫金诚征信有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411255152.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-05
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

传统的数据处理系统往往依赖于单次调用的业务逻辑,这种方式在处理大规模数据时显得效率低下,难以满足实时性和灵活性的需求

Benefits of technology

[0034] This application is a massive data decision-making engine system based on Java. The system includes a workspace module and an engine management module. The workspace module is connected to the engine management module; the engine management module includes a data import module and a strategy design module. module and policy deployment module; the workspace module is used to display the content in the engine management module; the engine management module is used to configure the data import module, the policy design module and the policy deployment module ; The data import module is used to import the database according to the configuration in the engine management module, and form a message queue for asynchronous processing; the strategy design module is used to form according to the configuration in the engine management module Multiple strategic decisions are loaded into the asynchronous processing message queue respectively, the data in the database is processed, and the processing results are obtained. The strategic decisions include a rule set, a decision tree, a decision table, and a score card. , one or more strategy methods in decision-making calculations and championship challenges; the strategy deployment module is used to dynamically adjust the CPU, memory and copy of the deployment container according to the configuration in the engine management module and kubernetes containerization technology number, and store the processing results in Elasticsearch, where the number of replicas represents the number of instances formed according to the policy decision. Use the workspace module to display the configuration process of the engine management module so that you can intuitively observe the configuration content. Use the engine management module to configure the data import module, policy design module and policy deployment module respectively. Importing the database in the data import module can also form a message queue for asynchronous processing so that multiple policy decisions formed by the policy design module are executed asynchronously. In addition, the Kubernetes containerization technology in the policy deployment module is used to make multiple policy decisions based on Dynamically adjust the CPU, memory and number of copies of the deployed container to facilitate load balancing of system resources. The number of copies can be dynamically adjusted according to traffic and load to optimize resource usage and response time. Next, after the processing results are stored in Elasticsearch, you can also use Elasticsearch's full-text search and analysis capabilities to efficiently index, query, and analyze the processing results. Make large-scale decisions on personal and enterprise-related information through databases and message queues to improve data processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118779115B_ABST
    Figure CN118779115B_ABST
Patent Text Reader

Abstract

The present application belongs to the field of data processing technology. A Java-based massive data decision engine system includes: a workspace module and an engine management module; the workspace module is used to display the content in the engine management module; the engine management module is used to configure the data import module, the policy design module and the policy deployment module; the data import module is used to import the database according to the configuration in the engine management module and form an asynchronously processed message queue; the policy design module is used to form multiple policy decisions according to the configuration in the engine management module, load them into the asynchronously processed message queue respectively, process the data in the database, and obtain the processing results; the policy deployment module is used to dynamically adjust the CPU, memory and number of copies of the deployed container according to the configuration in the engine management module and the kubercnetes containerization technology, and store the processing results in Elasticsearch.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular to a Java-based massive data decision engine system and method. Background Art

[0002] In today's data-driven business environment, enterprises and individuals are faced with the challenge of processing and analyzing large amounts of data. Traditional data processing systems often rely on single-call business logic, which is inefficient when processing large amounts of data and difficult to meet the requirements of real-time and flexibility. In order to solve these problems, it is particularly important to develop a decision engine system that can efficiently process and analyze massive amounts of data. Summary of the invention

[0003] Based on this, it is necessary to provide a Java-based massive data decision engine system to address the above technical issues.

[0004] In a first aspect, the present application provides a Java-based massive data decision engine system, the system comprising: a workspace module and an engine management module, the workspace module being connected to the engine management module; the engine management module comprising a data import module, a strategy design module and a strategy deployment module;

[0005] The workspace module is used to display the content in the engine management module;

[0006] The engine management module is used to configure the data import module, the strategy design module and the strategy deployment module;

[0007] The data import module is used to import the database according to the configuration in the engine management module and form a message queue for asynchronous processing;

[0008] The strategy design module is used to form multiple strategy decisions according to the configuration in the engine management module, load them into the asynchronous processing message queue respectively, process the data in the database, and obtain processing results, wherein the strategy decision includes one or more strategy methods of rule set, decision tree, decision table, score card, decision calculation and champion challenge;

[0009] The policy deployment module is used to dynamically adjust the CPU, memory and number of copies of the deployed container according to the configuration in the engine management module and utilize the kubercnetes containerization technology, and store the processing results in Elasticsearch, wherein the number of copies represents the number of container instances deployed to execute policy decisions.

[0010] Some practicable methods further include a data source management module, wherein the data source management module is connected to the workspace module;

[0011] The data source management module is used to adjust the database, and the adjustment of the database includes adding and deleting the database.

[0012] In some practicable methods, the data source management module has an Elasticsearch search engine unit, and the Elasticsearch search engine unit is used to store the processing results so that the Elasticsearch search engine unit can be used to call and record the processing results, generate engine reports, and export data.

[0013] In some practicable ways, the policy deployment module includes a call recording unit, an engine reporting unit, and a data exporting unit;

[0014] The call record unit is connected to the data source management module and is used to perform call record viewing through the Elasticsearch search engine unit;

[0015] The engine report unit is connected to the data source management module and is used to generate an engine report through the Elasticsearch search engine unit;

[0016] The data export unit is connected to the data source management module and is used to export data through the Elasticsearch search engine unit.

[0017] In some practicable ways, the policy deployment module further includes a release version unit;

[0018] The version publishing unit is used for version publishing and adjusts the workspace module and the engine management module according to the version publishing.

[0019] In some practicable ways, the data import module includes a database import unit and a kafka import unit;

[0020] The data import unit is used to add new data to the database by selecting fields in the database table;

[0021] The kafka import unit is used to parse and import the data of the database through the fields of the message body to form an asynchronously processed message queue.

[0022] In some implementable methods, the strategy design module is also used to use the result data of the first strategy decision as the input data of the second strategy decision in the canvas to form a series of the strategy decisions, wherein the multiple strategy decisions include at least the first strategy decision and the second strategy decision, and the first strategy decision and the second strategy decision are different strategy methods.

[0023] Some practicable methods further include a system management module, which is connected to the workspace module; the system management module includes a user management unit, a role management unit and a menu management unit;

[0024] The user management unit is used to add and delete relevant information of the user;

[0025] The role management unit is used to grant corresponding permissions to the users included in the user management unit so that the users have corresponding permissions in the engine management module;

[0026] The menu management unit is used to manage the addition, deletion, modification and query of the menu of the workspace module.

[0027] In a second aspect, the present application provides a Java-based massive data decision engine method, which is applied to the aforementioned Java-based massive data decision engine system, and the method includes:

[0028] In the case of displaying the content in the engine management module, configuring the data import module, the policy design module and the policy deployment module using the engine management module;

[0029] According to the configuration of the engine management module, the database is imported into the data import module, and a message queue for asynchronous processing is formed;

[0030] According to the configuration of the engine management module, a plurality of policy decisions are formed in the policy-related module, and the plurality of policy decisions are respectively loaded into the asynchronous processing message queue, the data in the database is processed, and the processing result is obtained;

[0031] According to the configuration in the engine management module, in the policy deployment module, the cpu, memory and number of copies of the deployed container are dynamically adjusted using the kubercnetes containerization technology, and the processing results are stored in Elasticsearch, wherein the number of copies represents the number of container instances deployed to execute policy decisions.

[0032] In a third aspect, the present application provides a computer storage medium, which implements the aforementioned system when executed by a processor.

[0033] Beneficial Effects

[0034] The present application discloses a massive data decision engine system based on Java, the system comprises a workspace module and an engine management module, wherein the workspace module is connected to the engine management module; the engine management module comprises a data import module, a policy design module and a policy deployment module; the workspace module is used to display the content in the engine management module; the engine management module is used to configure the data import module, the policy design module and the policy deployment module; the data import module is used to import a database according to the configuration in the engine management module and form an asynchronously processed message queue; the policy design module is used to load a plurality of policy decisions formed according to the configuration in the engine management module into the asynchronously processed message queue respectively, process the data in the database and obtain a processing result, wherein the policy decision comprises one or more policy methods in a rule set, a decision tree, a decision table, a scoring card, a decision calculation and a champion challenge; the policy deployment module is used to dynamically adjust the CPU, memory and number of copies of the deployment container according to the configuration in the engine management module and the kubernetes containerization technology, and store the processing result in Elasticsearch, wherein the number of copies represents the number of instances formed according to the policy decision. The workspace module is used to display the configuration process of the engine management module so that the configuration content can be observed intuitively. The engine management module is used to configure the data import module, the policy design module, and the policy deployment module respectively. The database is imported in the data import module, and a message queue for asynchronous processing can be formed so that multiple policy decisions formed by the policy design module can be executed asynchronously. In addition, the kubernetes containerization technology in the policy deployment module is used to dynamically adjust the CPU, memory, and number of copies of the deployment container according to multiple policy decisions in order to load balance system resources. The number of copies can be dynamically adjusted according to traffic and load to optimize resource usage and response time. Next, after the processing results are stored in Elasticsearch, the full-text search and analysis capabilities of Elasticsearch can be used to efficiently index, query, and analyze the processing results. Through the database and message queue, decisions are made on a large number of relevant information about individuals and enterprises to improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 The figure is a framework diagram of a Java-based massive data decision engine system in one embodiment.

[0037] Figure 2 The figure is a flow chart of a Java-based massive data decision engine system in one embodiment. DETAILED DESCRIPTION

[0038] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all couplings of one or more related listed items.

[0040] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0041] Some terms involved in this application are explained below to facilitate understanding of this application:

[0042] An engine is an application in a software system that processes data and makes decisions based on business rules, models, or algorithms.

[0043] The canvas refers to the operation area. When a program is selected in the operation area, it can be presented on the canvas, and the preset content can be displayed through the canvas.

[0044] Kafka is a distributed stream processing platform developed by the Apache Foundation, mainly used for high-throughput, low-latency data stream transmission and processing.

[0045] Kubernetes containerization technology, commonly known as Kubernetes, is an open source platform for automating the management, deployment, and expansion of containerized applications. It simplifies the lifecycle management of container applications by providing cluster management, load balancing, automated deployment, rolling updates, fault recovery, and other functions. Kubernetes makes large-scale operation and maintenance of container applications more efficient and reliable through its flexible architecture.

[0046] Pod, in Kubernetes, Pod is the smallest deployment unit, representing one or more containers (usually a single container) running in the cluster, which share storage and network resources.

[0047] Elasticsearch is an open source search engine built on Apache Lucene. It provides a distributed, multi-tenant full-text search engine and is widely used in log analysis, search engines, business analysis and other fields.

[0048] Prometheus is an open source monitoring and alerting tool suite that is usually used to record real-time time series data. It collects and stores metrics and provides a data query language (PromQL) to allow users to query and analyze this data.

[0049] Grafana is an open source data visualization and monitoring platform. It provides a rich user interface that allows users to display and analyze data through charts, panels, and dashboards.

[0050] Fluentd, is an open source data collector that allows unified data collection and consumption for better use of data. Fluentd is extensible, supports multiple plug-ins, and can be easily integrated with various data sources and target storage systems.

[0051] Logstash is an open source server-side data processing pipeline that can simultaneously collect, parse, and enrich log data, and then forward the data to a repository such as Elasticsearch.

[0052] Deployment is one of the key resource objects for implementing continuous deployment and operation and maintenance of applications in Kubernetes. Through Deployment, users can automate the application deployment process and ensure high availability of applications.

[0053] Service, in Kubernetes, is an API object that defines a way to access containerized applications, regardless of how the backend Pod changes. Service provides a single access interface for a group of Pods with the same functionality.

[0054] ConfigMap is an API object used to store configuration information in Kubernetes. It allows decoupling configuration data from application code, making application configuration more flexible and easier to manage.

[0055] YAML (YAML Ain't Markup Language) is a data serialization format for configuration files. It aims at data readability and is commonly used in the definition of Kubernetes resource objects.

[0056] Kibana, is an open source data analysis and visualization platform, often used with Elasticsearch. It is specifically designed to visualize Elasticsearch data and provides a user-friendly web interface to perform complex search, analysis, and visualization.

[0057] like Figure 1 and Figure 2 As shown, the first aspect of the present application provides a Java-based massive data decision engine system, the system comprising: a workspace module and an engine management module, the workspace module is connected to the engine management module; the engine management module comprises a data import module, a strategy design module and a strategy deployment module.

[0058] The workspace module is used to display the content in the engine management module. That is to say, after a user with permission clicks on the workspace module, he can enter the workspace module. The user can see the list of workspaces within the user's permission on this page, and the user can also add a new workspace. In this way, users with access rights can see the list of all engine management modules under the workspace on this page, and users can also view engine details and newly added engines. There can be a canvas in the workspace module so that corresponding operations can be performed in the canvas and displayed in the canvas. For example, operations such as displaying engine details and adding new engines in the canvas.

[0059] The engine management module is used to configure the data import module, the policy design module and the policy deployment module. The data import module, the policy design module and the policy deployment module can be accessed through the engine management module, and the data import module, the policy design module and the policy deployment module can be adjusted using the engine management module.

[0060] Specifically, the configuration of the data import module, policy design module and policy deployment module included in the engine management module can be understood as at least the parameter configuration of the data import module, policy design module and policy deployment module, so that the data import module, policy design module and policy deployment module can be executed under the configured parameters. Among them, parameter setting can refer to defining and adjusting the parameters required for the operation of each module.

[0061] The data import module is used to import the database according to the configuration in the engine management module and form a message queue for asynchronous processing.

[0062] Specifically, the data import module includes a database import unit and a kafka import unit.

[0063] The data import unit is used to add new additions to the database by selecting fields of the database table. That is, the database import unit directly interacts with the database. When a user or application selects fields of a specific table through a database query, the data import process is triggered. During this import process, it may be carried out synchronously and there may be a certain delay. The data import unit can be tightly coupled with the database, which also causes the import operation of the data import unit to depend on the structure and availability of the database.

[0064] The Kafka import unit is used to parse and import the data in the database through the fields of the message body to form an asynchronously processed message queue, that is, the data in the message queue is received through Kafka, and the data import process is triggered by sending the message to Kafka. In this way, once the message is sent to Kafka, it can be processed to form an asynchronously processed message queue, so that the distributed characteristics of Kafka provide good scalability.

[0065] It should be noted that the database import unit and the Kafka import unit can implement different processing mechanisms for the database to adapt to different data processing requirements. For example, the database import unit is suitable for batch processing and static data; the Kafka import unit is suitable for real-time processing and dynamic data. The data import unit is suitable for simple data import tasks; the Kafka import unit is suitable for complex scenarios that require high performance and real-time processing.

[0066] The strategy design module is used to form multiple strategy decisions according to the configuration in the engine management module, load them into the asynchronous processing message queue respectively, process the data in the database, and obtain the processing results. The processing results refer to the output obtained after the strategy design module applies a series of strategy decisions to the data in the database.

[0067] The strategic decision includes one or more strategic methods in rule sets, decision trees, decision tables, score cards, decision calculations, and champion challenges. It can be understood that the strategic methods such as decision tables, decision trees, rule sets, score cards, decision calculations, and champion challenges are all conventional methods, and this application does not improve them.

[0068] Specifically, under the parameters set by the engine management module, the strategy design module forms multiple strategy decisions according to user needs. For example, the strategy decision can be a strategy method in a rule set, a decision tree, a decision table, a scorecard, a decision calculation, and a champion challenge. That is to say, each strategy method forms a strategy decision. When there are multiple strategy decisions and they are sent to the data import module, the Kafka of the kafka import unit can be used to asynchronously process multiple strategy decisions, thereby improving data processing efficiency. In addition, the strategy design module is also used to use the result data of the first strategy decision as the input data of the second strategy decision in the canvas to form a series of the strategy decisions, wherein the multiple strategy decisions include at least the first strategy decision and the second strategy decision, and the first strategy decision and the second strategy decision are different strategy methods. The above method can be used to form and use multiple strategy decisions according to different needs, and improve the efficiency of data processing through the asynchronous processing mechanism of Kafka; support the series connection of strategy decisions to realize complex data processing logic.

[0069] It should be noted that when the result data of the first policy decision is used as the input data of the second policy decision, illustratively, the result data of the first policy decision is in JSON format, and the input data of the second policy decision requires CSV. Obviously, the result data of the first policy decision cannot be directly used as the input data of the second policy decision. In this case, the following steps are included:

[0070] Get all policy methods in policy decision;

[0071] According to all strategy methods, the input data format corresponding to each strategy method is obtained;

[0072] Obtaining a result data format of a first policy decision and an input data format of a second policy decision;

[0073] According to the input data format of the second policy decision, the result data format of the first policy decision is converted to obtain result data that conforms to the input data format of the second policy decision;

[0074] The result data conforming to the input data format of the second policy decision is input into the second policy decision.

[0075] It should be noted that when converting the data format, the following steps are included:

[0076] According to the input data format corresponding to each strategy method, an intermediate data model is constructed; wherein the intermediate data model contains the data fields required by all strategy methods and defines a unified data type and format.

[0077] In the intermediate data model, a data mapping rule is constructed for each strategy method to obtain an intermediate data model mapping rule;

[0078] According to the intermediate data model mapping rules, the result data of the first policy decision is mapped to obtain data of a unified data type and format output by the intermediate data model; wherein, after the mapping is performed in the intermediate data model, the result data will be converted into the format of the intermediate data model, which can be recognized by the second policy decision.

[0079] The data of uniform data type and format output by the intermediate data model is used as input data of the second policy decision and input into the second policy decision.

[0080] It should also be noted that an adapter is configured for each policy method, so that, illustratively, policy method A uses the JSON format. Policy method B uses the XML format. Two adapters are written for policy method A, one adapter converts JSON to an intermediate data model, and the other adapter converts the intermediate data model back to JSON. Two adapters are written for policy method B, one adapter converts XML to an intermediate data model, and the other adapter converts the intermediate data model to XML. An adapter refers to a software design pattern that converts one format into another, enabling two incompatible interfaces to work together, and is usually used to convert the interface of a class into another interface expected by the client.

[0081] The above method simplifies complex data exchange scenarios, especially in the case of multiple policy methods. Although each policy method may require two adapters, overall, this architecture reduces the dependency between systems, simplifies data management, and improves overall maintainability and scalability.

[0082] The policy deployment module is used to dynamically adjust the CPU, memory and number of copies of the deployment container according to the configuration in the engine management module and the kubernetes containerization technology, and store the processing results in Elasticsearch, wherein the number of copies indicates the number of container instances deployed to execute the policy decision. In addition, users can also customize the CPU, memory and number of copies of the project deployment container as needed.

[0083] It should be noted that in the dynamic adjustment of cpu, memory and number of replicas of deployed containers by Kubernetes containerization technology, Kubernetes allows resource requests and resource limits to be set for each container. Resource requests tell the Kubernetes system the minimum amount of resources required by the container, while resource limits are the maximum amount of resources that the container can use. By modifying the resource requests and limits in the deployment configuration, you can redefine the resource requirements of the container and apply these changes.

[0084] Dynamic adjustment can be horizontal dynamic expansion (Horizontal Pod Autoscaler, HPA) and vertical dynamic expansion (Vertical Pod Autoscaler, VPA), etc. In horizontal dynamic expansion, Kubernetes' horizontal automatic expansion function can automatically adjust the number of Pod replicas based on CPU usage or other selected metrics. HPA monitors the usage of Pods and increases or decreases the number of replicas when needed. In vertical automatic expansion, the resource request of Pods can be automatically adjusted to more accurately match the resource requirements of applications. It will analyze the historical resource usage of Pods and recommend appropriate CPU and memory requests.

[0085] Specifically, it is used to dynamically adjust the CPU, memory and number of copies of the deployed container according to the configuration in the engine management module by using the kubernetes containerization technology, and store the processing results in Elasticsearch, including the following steps:

[0086] Determine the configuration file of the kubernetes resource object according to the requirements of deploying containers;

[0087] The configuration files of kubernetes resource objects include Deployment, Service, and ConfigMap. Specifically, a Deployment YAML file is created, which contains the image of the application container, the initial number of replicas, resource requests, and restrictions. ConfigMap is used to store non-sensitive configuration information, and Secrets is used to store sensitive information such as database passwords.

[0088] According to the configuration file of the kubercnetes resource object, deploy the container application to the Kubernetes cluster and establish the initial running state of the deployed container application;

[0089] The deployment container is applied to the Kubernetes cluster to ensure that the application service is available. For example, use kubectl apply -f<deployment-file.yaml> Deploy the application. Verify that the Deployment, Pods, and related Services are started normally.

[0090] Based on the initial running status of the deployed container application, dynamically adjust the number of Pod copies, resource usage and responsiveness according to the actual load of the deployed container application;

[0091] Among them, resources can include computing resources and storage resources. Computing resources are physical or virtual resources used to run containerized applications in the Kubernetes cluster. Although storage resources are usually provided by external storage systems in containerized environments, in some cases, such as when using EmptyDir or PersistentVolume, the allocation and use of storage resources also need to be considered. Specifically, according to the load characteristics of the application, define the rules and thresholds of HPA and write the HPA YAML file. Use kubectl apply -f<hpa-file.yaml> The command applies the HPA configuration, enabling the system to automatically adjust the number of Pods based on CPU utilization or other metrics.

[0092] According to the configuration file of the kubercnetes resource object, deploy the container application to the Kubernetes cluster, establish the initial running state of the deployed container application, and dynamically adjust the number of Pod copies according to the actual load of the deployed container application, adjust resource usage and responsiveness, and integrate Elasticsearch.

[0093] Among them, integrating Elasticsearch can store the processing results of the application in Elasticsearch, realizing data persistence and efficient retrieval. Specifically, ensure that the Elasticsearch service is available in the Kubernetes cluster. Integrate the Elasticsearch client in the application to realize the ability to index the processing results into Elasticsearch.

[0094] Integrate Elasticsearch for monitoring and log management;

[0095] Among them, monitor the performance of applications and Kubernetes clusters, collect logs, and ensure the stability and maintainability of the system. Specifically, integrate Prometheus and Grafana for system monitoring, configure Prometheus to capture and visualize key performance indicators. Configure a log collection system such as Fluentd or Logstash to send logs to Elasticsearch, and use Kibana for log analysis.

[0096] It should also be noted that the policy deployment module is also used for version release, and the workspace module and the engine management module are adjusted according to the version release. Among them, the policy deployment module is a part of the system, which is responsible for applying the designed policy decisions to data decisions and ensuring that they can be executed correctly. Version release refers to updating the new version of the software or system to a user-accessible state. This may include new features, improvements or bug fixes. Through version release, the policy deployment module can manage different versions of policy designs to ensure that it is possible to roll back to the old version, or gradually launch the new version. In addition, the workspace module refers to the environment in which users interact with the system. When a new version is released, the workspace module can be adjusted, such as adjusting the canvas of the workspace, updating the user interface, functions or data display methods.

[0097] In some embodiments, a Java-based massive data decision engine system further includes a data source management module. The data source management module is connected to the workspace module; the data source management module is used to adjust the database, and the adjustment of the database includes adding and deleting the database.

[0098] Specifically, users can add databases in the data source management module as needed, supporting MySQL, PostgreSql, Hive and other types of databases, so that the data import module can call the database in the data source management module.

[0099] It should be noted that the data source management module also has an Elasticsearch search engine unit, which is used to store the processing results so that the Elasticsearch search engine unit can be used to call and record the processing results, generate engine reports, and export data.

[0100] It should also be noted that the Elasticsearch search engine unit is used to call the processing results stored in Elasticsearch, which allows the processing results to be indexed and quickly retrieved. Furthermore, call record viewing, Elasticsearch can store call records, allowing users to view the usage of the engine, including information such as the number of calls and time. Engine report generation, using the aggregation function of Elasticsearch, you can generate reports, such as total call volume, success rate, response time and other statistical information. Data export, users may need to export processing results from Elasticsearch for further analysis or backup.

[0101] In some embodiments, a Java-based massive data decision engine system further includes a system management module, which is connected to the workspace module; the system management module includes a user management unit, a role management unit and a menu management unit.

[0102] The user management unit is used to add and delete the relevant information of the user.

[0103] The role management unit is used to grant corresponding permissions to the users included in the user management unit so that the users have corresponding permissions in the engine management module.

[0104] The menu management unit is used to manage the addition, deletion, modification and query of the menu of the workspace module.

[0105] Example 1

[0106] The user logs in to the system and jumps to the system homepage after the password is verified. The user can see the two menu information of workspace and data source management on this page. The user can add databases in the data source management, which supports mysql, PostgreSql, hive and other types of databases. User permissions and engine permissions can be set.

[0107] When the user clicks on the workspace, he can see the list of workspaces within the user's permissions on this page. The user can also add a workspace and set access permissions. When entering the workspace, the user can see the list of all engines under the workspace on this page. The user can also view engine details and add new engines. Click the engine name to enter the engine details. The user can see the data import, policy design, and policy deployment menus on this page. When entering the data import, the user can add the field information required for the logical judgment of the engine component on this page. The user can import through the database and message components such as Kafka. The database is added by selecting the fields of the database table. The message components such as Kakfa are imported by parsing the fields of their message bodies.

[0108] Enter the strategy design page, where users can add decision tables, decision trees, rule sets, scorecards, decision calculations, champion challenges and other components, and save the components in series in the canvas. Enter the strategy deployment page, where users can see the release version, call records, engine reports, data export menu, and data export. Users can add engine result export methods on the page, which supports message component export such as kafka and database export. Enter the release version, where users can release versions on the message page. This function is implemented through kubernetes containerization technology, and users can customize the cpu, memory and number of copies of the project deployment container. Enter the call page, where users can see the list of historical call records of the engine. Enter the engine report page, where users can see the statistics of the engine, such as the total number of calls, the number of successful and failed calls, the response time, the total number of calls trend statistics, and the month-on-month / year-on-year growth rate. Entering the system management, users can see user management, role management, and menu management on this page. Entering the user management, users can add and modify user information and query the user list on this page. Entering the menu page, users can add and modify menu information and view the menu list on this page. Entering the role management page, users can add and modify role information and view the user list on this page.

[0109] Example 2

[0110] The user's related operations in the workspace, policy deployment, and policy design are stored in the MySQL database. The engine's processing results and related information are stored in Es (Elasticsearch). Users can view Es data in call records and engine reports.

[0111] System functional module design

[0112] (1) User login interface: It is the entrance for users to access the system. Users can log in to the system through the visual login page.

[0113] (2) User management unit: Administrators can use this page to add new users and modify user information, user organizations, and roles.

[0114] (3) Role management unit: Administrators can manage all role information and the roles of all users, thereby controlling the user's permissions in the system;

[0115] (4) Menu management unit: The administrator can manage the addition, deletion, modification and query of all menus in the system;

[0116] (5) Data source management module: Administrators can add, delete, modify and query the data source information of this system.

[0117] (6) Data import module: Users can import data through databases and add engine logic fields through message components such as Kafka;

[0118] (7) Strategy design module: Users can add, delete, modify and check components such as rule sets, decision trees, decision tables, scorecards, decision calculations, and champion challenges;

[0119] (8) Data export unit: Users can add, delete, modify and query the export path of engine results;

[0120] (9) Call record unit: Users can query the history list and report details of the engine report.

[0121] (10) Engine report unit: Users can query the total number of engine calls, number of successful and failed calls, response time, total number of calls trend statistics, month-on-month / year-on-year growth rate and other statistical information.

[0122] (11) Release version unit: After completing the addition of new components, users can deploy the engine.

[0123] In summary, the application provides a Java-based massive data decision engine system to achieve rapid acquisition of credit subject information. It is based on the springBoot model, which makes it easy to develop the system. Its low cost and easy operation make it widely used. The underlying data is stored in elasticsearch, which provides near real-time search and analysis for all types of data. Whether it has structured or unstructured text, digital data or geospatial data, Elasticsearch can efficiently store and index in a way that supports fast search, and can go beyond simple data retrieval and aggregate information to discover trends and patterns in the data. As the amount of data and queries grows, the distributed nature of Elasticsearch enables deployment to grow seamlessly. Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. It is very convenient to enable large amounts of data to have the ability to search, analyze and explore. Taking full advantage of the horizontal scalability of Elasticsearch can make data more valuable in the production environment. The engine deployment uses Kubernetes containerization technology to deploy projects, and can dynamically adjust the resource situation of project deployment, greatly saving resources and providing flexible and changeable project deployment.

[0124] The present application is a java-based massive data decision engine system, which uses the engine management module to configure the strategy design module to form multiple strategy decisions, and then applies the multiple strategy decisions in the data import module, calls the corresponding data execution, and uses the asynchronous processing method to realize the synchronous processing of multiple strategy decisions. In order to cope with the synchronous processing of multiple strategy decisions, the present application introduces the kubernetes containerization technology, and uses the kubernetes containerization technology to adjust the cpu, memory and number of copies of the deployment container, so that each strategy decision can have the corresponding cpu and memory to execute, thereby ensuring the execution efficiency. Next, Elasticsearch is used to store the results, which is convenient for users to call the processing results in Elasticsearch, and in Elasticsearch, it is very convenient to make a large amount of data have the ability to search, analyze and explore. Through the above method, the call of multiple businesses is realized, and the work efficiency is improved. It has the function of developing concise and efficient, and making decisions on the relevant information of individuals and enterprises in large quantities through databases and MQ.

[0125] In a second aspect, the present application provides a Java-based massive data decision engine method, which is applied to the aforementioned Java-based massive data decision engine system, and the method includes:

[0126] In the case of displaying the content in the engine management module, configuring the data import module, the policy design module and the policy deployment module using the engine management module;

[0127] According to the configuration of the engine management module, the database is imported into the data import module, and a message queue for asynchronous processing is formed;

[0128] According to the configuration of the engine management module, a plurality of policy decisions are formed in the policy-related module, and the plurality of policy decisions are respectively loaded into the asynchronous processing message queue, the data in the database is processed, and the processing result is obtained;

[0129] According to the configuration in the engine management module, in the policy deployment module, the cpu, memory and number of copies of the deployed container are dynamically adjusted using the kubercnetes containerization technology, and the processing results are stored in Elasticsearch, wherein the number of copies represents the number of container instances deployed to execute policy decisions.

[0130] In a third aspect, the present application provides a computer storage medium, which implements the aforementioned system when executed by a processor.

[0131] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through computer storage media, and the computer storage medium can be stored in a non-volatile computer-readable storage medium, which, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0132] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0133] The protection scope of the present disclosure is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A Java-based massive data decision engine system, characterized in that: The system includes: A workspace module and an engine management module, wherein the workspace module is connected to the engine management module; The engine management module includes a data import module, a strategy design module and a strategy deployment module; The workspace module is used to display the content in the engine management module; The engine management module is used to configure the data import module, the strategy design module and the strategy deployment module; The data import module is used to import the database according to the configuration in the engine management module and form a message queue for asynchronous processing; The strategy design module is used to form multiple strategy decisions according to the configuration in the engine management module, load them into the asynchronous processing message queue respectively, process the data in the database, and obtain processing results, wherein the strategy decision includes one or more strategy methods of rule set, decision tree, decision table, score card, decision calculation and champion challenge; The policy deployment module is used to dynamically adjust the CPU, memory and number of copies of the deployed container according to the configuration in the engine management module and using the kubernetes containerization technology, and store the processing results in Elasticsearch, wherein the number of copies represents the number of container instances deployed to execute the policy decision; wherein, The strategy design module is further used to use the result data of the first strategy decision as the input data of the second strategy decision in the canvas to form a series of the strategy decisions, wherein the multiple strategy decisions include at least the first strategy decision and the second strategy decision, and the first strategy decision and the second strategy decision are different strategy methods; wherein, When the result data of the first policy decision is used as the input data of the second policy decision, the following steps are included: Get all policy methods in policy decision; According to all strategy methods, the input data format corresponding to each strategy method is obtained; Obtaining a result data format of a first policy decision and an input data format of a second policy decision; According to the input data format of the second policy decision, the result data format of the first policy decision is converted to obtain result data that conforms to the input data format of the second policy decision; The result data conforming to the input data format of the second policy decision is input into the second policy decision.

2. The Java-based massive data decision engine system according to claim 1, characterized in that: It also includes a data source management module, which is connected to the workspace module; The data source management module is used to adjust the database, and the adjustment of the database includes adding and deleting the database.

3. The Java-based massive data decision engine system according to claim 2 is characterized in that: The data source management module has an Elasticsearch search engine unit, and the Elasticsearch search engine unit is used to store the processing results so that the Elasticsearch search engine unit can be used to call and record the processing results, generate engine reports, and export data.

4. The Java-based massive data decision engine system according to claim 3 is characterized in that: The policy deployment module includes a call recording unit, an engine reporting unit and a data exporting unit; The call record unit is connected to the data source management module and is used to perform call record viewing through the Elasticsearch search engine unit; The engine report unit is connected to the data source management module and is used to generate an engine report through the Elasticsearch search engine unit; The data export unit is connected to the data source management module and is used to export data through the Elasticsearch search engine unit.

5. The Java-based massive data decision engine system according to claim 4, characterized in that: The policy deployment module also includes a version publishing unit; The version publishing unit is used for version publishing and adjusts the workspace module and the engine management module according to the version publishing.

6. The Java-based massive data decision engine system according to claim 1, characterized in that: The data import module includes a database import unit and a kafka import unit; A data import unit, used to add new data to the database by selecting fields in the database table; The kafka import unit is used to parse and import the data of the database through the fields of the message body to form an asynchronously processed message queue.

7. The Java-based massive data decision engine system according to claim 1, characterized in that: It also includes a system management module, which is connected to the workspace module; the system management module includes a user management unit, a role management unit and a menu management unit; The user management unit is used to add and delete relevant information of the user; The role management unit is used to grant corresponding permissions to the users included in the user management unit so that the users have corresponding permissions in the engine management module; The menu management unit is used to manage the addition, deletion, modification and query of the menu of the workspace module.

8. A Java-based massive data decision engine method, characterized in that: A Java-based massive data decision engine system applied to any one of claims 1-7, the method comprising: In the case of displaying the content in the engine management module, configuring the data import module, the policy design module and the policy deployment module using the engine management module; According to the configuration of the engine management module, the database is imported into the data import module, and a message queue for asynchronous processing is formed; According to the configuration of the engine management module, a plurality of policy decisions are formed in the policy-related module, and the plurality of policy decisions are respectively loaded into the asynchronous processing message queue, the data in the database is processed, and the processing result is obtained; According to the configuration in the engine management module, in the policy deployment module, the cpu, memory and number of copies of the deployed container are dynamically adjusted using the kubernetes containerization technology, and the processing results are stored in Elasticsearch, wherein the number of copies represents the number of container instances deployed to execute policy decisions.

9. A computer storage medium, characterized in that When the computer storage medium is executed by a processor, the system of any one of claims 1 to 7 is implemented.