A method for automatically solving data analysis problems based on knowledge search

Abstract

The present invention relates to a method for automatically solving data analysis problems based on knowledge search, which specifically includes: step 1, using neo4j graph database as a carrier to construct a knowledge graph; step 2, designing an analysis problem definition, and converting a specific analysis problem into a json object; step 3, building a knowledge search module, searching for a solution in the knowledge graph according to the problem json object, and searching for its solution using the knowledge graph as a search space; step 4, building an automatic calculation module, automatically parsing the solution into an executable program, and displaying the analysis result based on a visualization component; step 5, solidifying and restoring a process flow program; for the process flow program generated in step 4, solidifying it into a json object file that complies with the PMML standard; loading a json file that complies with the PMML standard, and reconstructing the process flow program in memory. The present invention avoids the tedious work of developing an analysis program and improves the efficiency and intelligence level of data analysis.

Description

Technical Field

[0001] The present invention relates to a visualization implementation method, and in particular to an automatic solution method for data analysis problems based on knowledge search. Background Art

[0002] In data analysis problems with rail equipment as the main body, it is difficult to effectively combine business domain knowledge and data analysis algorithms, and they are in a relatively fragmented state. For analysts, the collected original data sets are highly coupled with the business domain, and the corresponding feature engineering and data processing require the support of relevant business knowledge; for business personnel, when using existing analysis software to process and analyze the original data, they need to have corresponding programming capabilities and have a corresponding understanding of the meaning of each parameter of the algorithm model, so that the analysis purpose can be achieved by configuring the parameters.

[0003] Due to factors such as the wide range of professions, it is difficult to require staff to be familiar with relevant business knowledge and have the ability to perform programming analysis. At present, data analysis problems for rail equipment are mainly solved based on descriptive analysis methods, namely "problem->data->model->analysis->results", that is: business personnel give guidance on data processing and feature engineering for the original data based on their own domain knowledge, clarify the expected results and specify the analysis model and input and output. Analysts process data based on preset processes (requirements), configure relevant models for fitting, and draw visual images as results. In this solution process, business personnel and analysts need to communicate repeatedly, which requires a lot of manpower and time costs; at the same time, the models and processing pre-selected based on business knowledge and experience are often not the optimal solution, which may affect the accuracy of data analysis. Summary of the invention

[0004] The purpose of the present invention is to solve the current separation problem between business fields and analysis algorithms, thereby optimizing the selection of solutions and improving solution efficiency, and solving the problem of the difficulty in combining business knowledge and analysis technology.

[0005] The technical solution of the present invention is to provide a method for automatically solving data analysis problems based on knowledge search, which is characterized by specifically comprising the following steps:

[0006] Step 1: Use the neo4j graph database as a carrier to build a knowledge graph;

[0007] The knowledge graph includes three parts: problem type model, analysis data set model, and processing node model. The problem type model and analysis data set model are designed based on the specific rail transit equipment and maintained by business personnel.

[0008] Step 2: Design the analysis problem definition and convert the specific analysis problem into a JSON object;

[0009] Step 3: Build a knowledge search module, generate corresponding search strategies based on the problem json object, use the knowledge graph as the search space to search for solutions that match the problem, and display and interact in a visual interface;

[0010] Step 4: Build an automatic calculation module to automatically parse the solution into an executable program and display the analysis results based on the visualization component;

[0011] Step 5: Solidify and restore the processing flow program, solidify the processing flow program generated in step 4 into a json object file that complies with the PMML standard;

[0012] Furthermore, in step 1:

[0013] The problem type model is divided into four types: descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, among which:

[0014] Descriptive analysis problems include two categories: state description and state evaluation;

[0015] Diagnostic analysis problems include four categories: time series correlation, comparison and screening of influencing factors, health diagnosis, and multivariate correlation;

[0016] Predictive analysis problems include regression problems and alarm prediction;

[0017] Decision analysis problems involve finding the optimal solution.

[0018] Furthermore, in step 2: for the rail equipment as the analysis object, the complete description of the analysis problem needs to include the analysis subject, problem description, known conditions, constraints, solution and expected results.

[0019] Furthermore, in step 3:

[0020] Step 3.1, read and parse the specific analysis problem json object, and initialize the solution search control strategy;

[0021] Step 3.2: Solve the analysis problem based on the search control strategy initialized in step 3.1 using the knowledge graph as the search space;

[0022] Step 3.3, repeat step 3.2 until the current node attribute in the control strategy is the same as the target node attribute;

[0023] Step 3.4: Screen the alternative solutions obtained in step 3.3 according to the functional requirements of the solution in the control strategy, and eliminate the solutions with incomplete functions; if there are multiple alternative solutions with complete functions, introduce the shortest path algorithm to further evaluate the alternative solutions, and select the solution with the shortest path and the lowest implementation cost as the search result; if there is no alternative solution with complete functions, select the solution with the most complete functions as the search result;

[0024] Step 3.5, display the search results obtained in step 3.4 in the form of nodes and directed relationships combined on a visual interactive page;

[0025] Step 3.6: Import the solution process finally determined in the visual interactive interface in step 3.5 in the form of a json object.

[0026] Furthermore, in step 4:

[0027] Step 4.1, read the solution json object finally generated in step 3, parse the processing nodes in the solution, and generate the corresponding processing component program; import the json object of the processing node into the pre-set program template, automatically generate the corresponding processing code based on the statement splicing method, and compile it in the memory to generate the corresponding component program;

[0028] Step 4.2: Assemble the component programs generated in step 4.1 into the computing engine pipeline in the form of a workflow according to the sequence of processing nodes in the solution; the last component in the pipeline acts as an estimator, and the other components work as converters, and the output result is the prediction result of the last component;

[0029] Step 4.3: The output result is json format data that is compatible with the visualization component. Import the output result into the corresponding visualization component to directly generate a visualization analysis chart to display the data.

[0030] Furthermore, in step 5:

[0031] Step 5.1: Solidify the processing flow program. Import the processing flow program in step 4 into the PMMLpipeline method of the sklearn library. Record the model structure, model parameters, and hyperparameters of the configuration model in the order of each node in the process based on the PMML standard, and solidify them in the form of a json file.

[0032] Step 5.2: The processing flow program is restored. Based on the from_file method in the pypmml.Model library, the PMML standard file is read and parsed, and the processing flow program solidified in the file is reloaded into the memory for use.

[0033] The beneficial effects of the present invention are:

[0034] (1) The present invention proposes a method for automatically solving data analysis problems based on knowledge search, using the knowledge graph as a bridge to solve the problem of the difficulty in combining business knowledge and analysis technology. Business personnel maintain business domain-related knowledge in the knowledge graph by defining problem models and data set models, and analysts maintain analysis technology-related methods in the knowledge graph by defining processing models. The model nodes in the knowledge graph are connected by directed relationships with semantic information, realizing the combination of "problem-data-analysis technology" and bridging the gap between business knowledge and analysis technology.

[0035] (2) For a given analysis problem, the present invention uses the knowledge graph as the search space, implements the knowledge search module through programming, dynamically adjusts the search direction according to the analysis problem and the relationship semantics, and gives the optimal solution in combination with the shortest path algorithm. Compared with the traditional analysis method, the present invention only needs to provide the analysis problem json object to automatically obtain the solution to the matching problem, which simplifies the analysis process design process and also provides support for the interpretability of the processing flow.

[0036] (3) The programming of the present invention implements an automatic calculation module, automatically analyzes and executes the solution provided in (2), and presents the analysis results in the form of visual charts; compared with traditional analysis methods, the present invention avoids the tedious work of developing analysis programs and improves the efficiency and intelligence level of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0038] Figure 2 The knowledge graph for the basic algorithm set is:

[0039] Figure 3 Set knowledge graphs for machine learning algorithms;

[0040] Figure 4 It is the knowledge graph of indicator algorithm set;

[0041] Figure 5 A knowledge graph for data processing algorithm sets;

[0042] Figure 6 It is a schematic diagram of the knowledge search process;

[0043] Figure 7 Interactive UI interface display for solution;

[0044] Figure 8 It is the flow chart of automatic calculation module;

[0045] Fig. 9Provides a visual display interface for Vegalite; DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solution of the present application, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] This embodiment provides a method for automatically solving data analysis problems based on knowledge search, which is used to solve the current separation problem between business fields and analysis algorithms, optimize the selection of solution solutions and improve the solution efficiency, and specifically includes the following steps:

[0048] Step 1: Use the neo4j graph database as a carrier to build a knowledge graph;

[0049] The knowledge graph mainly includes three parts: problem type model, analysis data set model, and processing node model. Among them, the problem type model and analysis data set model are designed based on the specific rail transit equipment and maintained by business personnel.

[0050] The problem type model is divided into four types: descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, among which:

[0051] Descriptive analysis problems include two categories: state description and state evaluation;

[0052] Diagnostic analysis problems include four categories: time series correlation, comparison and screening of influencing factors, health diagnosis, and multivariate correlation;

[0053] Predictive analysis problems include regression problems and alarm prediction; decision analysis includes optimal solutions.

[0054] The analysis data set model includes four parts: analysis data set definition, analysis data set columns, data structure characteristics, and data quality characteristics. According to the cycle status, it is divided into categories such as factory data, test data, fault data, maintenance data, and scrapped data.

[0055] At the same time, the analytical data set model also maintains an indicator model for evaluating the original data, which is divided into four categories: time dimension, location dimension, shape dimension and algorithm dimension. Among them, the indicator model serves as a bridge to connect the relevant analytical data set columns and processing nodes in the form of relationships in the graph database.

[0056] The processing node models are divided into three categories: basic algorithm set, basic rule set and visualization component;

[0057] Among them: the basic algorithm set maintains the applicable scenarios, data set assumptions, and meanings of the corresponding parameters of the corresponding algorithm program in the form of nodes, and maintains the configuration required to generate the executable program and the corresponding calling statements in the form of json; the basic algorithm set is divided into three categories according to the different types of algorithms: machine learning algorithm set, indicator algorithm set, and data processing algorithm set, and defines the unique properties of its own category based on the definition of the basic algorithm set attributes; the embodiment uses the neo4j graph database as the data source and combines the InteractiveGraph control to display the knowledge graph. Figure 2 is the basic algorithm set model, Figure 3 is a machine learning algorithm set model, Figure 4 is the indicator algorithm set model, Figure 5 It is the knowledge graph corresponding to the data processing algorithm set model.

[0058] The basic rule set maintains the underlying rules such as logical expressions, logic gates, and operators;

[0059] The visualization component set maintains the methods, tools, parameters required for instantiation, and corresponding calling rules for implementing the visualization process, such as Vegalite, Tableau, PowerBI, Gephi, etc.

[0060] Nodes are connected by maintaining relationships in the knowledge graph. The relationship categories between nodes include inheritance, dependency, and connection. Relevant semantic information such as name, label, description, and service type are added to the relationship.

[0061] It should be noted that when migrating the knowledge graph for urban rail train wheelset components to other rail transit equipment, it is only necessary to replace the business-related knowledge graph according to needs, and the relevant algorithms and processing nodes can continue to be used.

[0062] Step 2: Design the analysis problem definition and convert the specific analysis problem into a JSON object;

[0063] For rail equipment as the analysis object, the complete description of the analysis problem needs to include the analysis subject, problem description, known conditions (problem type, original data set, problem environment, etc.), constraints, solution and expected results, etc., which can be formally expressed as:

[0064] AP=(As,description,resource,restriction,method,goal)

[0065] in:

[0066] AP stands for Given Data Analysis Problem;

[0067] As represents the subject of analysis;

[0068] description represents the text description of the analysis problem;

[0069] resource represents the set of available resources in the initial state, i.e., known conditions;

[0070] method represents a solution to a problem;

[0071] Restriction represents the set of restrictions on solving the problem;

[0072] Goal represents the expected goal.

[0073] Taking the health status trend analysis problem of urban rail train wheelset components as an example, the text description of the problem is: "Perform data analysis on the historical health data of wheelset components, establish relevant models to describe the changing trend of wheelset health status, and thus predict the future health status of wheelsets."

[0074] The analysis problem is formalized based on the analysis problem definition, where the analysis subject object (As) is the wheelset component (Wheelset); the problem text description (description) is the original text; the known condition (resource) defines the problem type as predictive analysis (predictive_analysis) and specifies the analysis data set path; the restriction (restriction) is none; the solution (method) is none; the expected goal (goal) is predictive analysis (prediction) and visualization (visualization). The corresponding json object is generated in the form of key-value pairs.

[0075] Step 3: Build a knowledge search module, generate a corresponding search strategy based on the problem json object, use the knowledge graph as the search space to search for solutions that match the problem, and display and interact in a visual interface.

[0076] The process is shown as follows Fig. 9 As shown, the specific implementation steps are as follows:

[0077] Step 3.1, read and parse the specific analysis problem json object, and initialize the solution search control strategy. Extract the problem type attributes in the known conditions, find the problem type node in the knowledge graph, set it as the current node of the knowledge search algorithm, and read the solution configuration corresponding to the problem type in the node; extract the expected target attributes, determine the target node of the search algorithm, and the functional requirements for the final search results.

[0078] Step 3.2: Solve the analysis problem based on the search control strategy initialized in step 3.1 using the knowledge graph as the search space. Read the current node name in the control strategy, search for the corresponding node in the knowledge graph, and store the node name in the solution attribute in the control strategy in the form of a list element; search for the relationship connected to the current node, read the semantic information in the relationship (relationship type, relationship direction), and if it matches the required semantics, search for the node at the other end of the relationship, read the node name attribute and update the current node attribute assigned to the search control strategy;

[0079] Step 3.3, repeat step 3.2 until the current node attribute in the control strategy is the same as the target node attribute; if there are multiple next nodes that meet the conditions at the same time during the process, the solution is split to form multiple groups of alternative solutions;

[0080] Step 3.4: Screen the alternative solutions obtained in step 3.3 according to the functional requirements of the solution in the control strategy, and eliminate the solutions with incomplete functions; if there are multiple alternative solutions with complete functions, introduce the shortest path algorithm to further evaluate the alternative solutions, and select the solution with the shortest path and the lowest implementation cost as the search result; if there is no alternative solution with complete functions, select the solution with the most complete functions as the search result;

[0081] Step 3.5: Display the search results obtained in step 3.4 in the form of nodes and directed relationships in a visual interactive page, where Figure 7 As shown in the figure, nodes represent processing nodes in the solution, and directed relationships show the flow direction of data. In this interface, you can modify the node configuration by right-clicking the node, or add or remove processing nodes in the process by dragging and dropping.

[0082] Step 3.6: Import the solution process finally determined in the visual interactive interface in step 3.5 in the form of a json object.

[0083] Step 4: Build the automatic calculation module. The process is as follows: Figure 8 As shown, the solution is automatically parsed into an executable program, and the analysis results are displayed based on the visualization component;

[0084] The specific implementation steps are as follows:

[0085] Step 4.1, read the solution json object finally generated in step 3, parse the processing nodes in the solution, and generate the corresponding processing component program; import the json object of the processing node into the preset program template, automatically generate the corresponding processing code based on the statement splicing method, and compile it in the memory to generate the corresponding component program; the executable component needs to adapt to the program requirements of the computing engine. In the embodiment, the component program needs to encapsulate the training (fit), data transmission (transforme) and prediction (predict) methods, so as to ensure the use and transmission of data by the component in the pipeline;

[0086] Step 4.2: Assemble the component programs generated in step 4.1 into the computing engine pipeline in the form of a workflow according to the sequence of processing nodes in the solution; the last component in the pipeline acts as an estimator, and the other components work as converters, and the output result is the prediction result of the last component;

[0087] Step 4.3: The output result is json format data adapted to the visualization component. Import the output result into the corresponding visualization component to directly generate a visualization analysis chart to display the data, such as Figure 8 As shown, the left side is the output json data, and the right side is the chart automatically generated based on the Vegalite-Editor component.

[0088] Step 5, solidification and recovery of the processing flow program, mainly includes two technical points: for the processing flow program generated in step 4, convert it into a json file that conforms to the PMML standard for storage; load the json file that conforms to the PMML standard, and rebuild the processing flow program in memory;

[0089] The specific implementation steps are as follows:

[0090] Step 5.1, solidify the processing flow program: import the processing flow program in step 4 into the PMMLpipeline method of the sklearn library, record the model structure, model parameters and hyperparameters of the configuration model in the order of each node in the process based on the PMML standard, and solidify them in the form of a json file;

[0091] Step 5.2, recovery of processing flow program: Taking the python environment in the embodiment as an example, the PMML standard file can be read and parsed based on the from_file method in the pypmml.Model library, and the processing flow program solidified in the file can be reloaded into the memory for use.

[0092] Although the principles of the present invention are described in detail above in conjunction with the preferred embodiments of the present invention, those skilled in the art should understand that the above embodiments are merely explanations of the exemplary implementations of the present invention, and are not intended to limit the scope of the present invention. The details in the embodiments do not constitute limitations on the scope of the present invention, and any obvious changes such as equivalent transformations, simple replacements, etc. based on the technical solution of the present invention, without departing from the spirit and scope of the present invention, fall within the protection scope of the present invention.

Claims

1. A method for automatically solving data analysis problems based on knowledge search, characterized in that: The specific steps include: Step 1: Use the neo4j graph database as a carrier to build a knowledge graph; The knowledge graph includes three parts: problem type model, analysis data set model, and processing node model. The problem type model and analysis data set model are designed based on the specific rail transit equipment and maintained by business personnel. The problem type model is divided into four types: descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, among which: Descriptive analysis problems include two categories: state description and state evaluation; Diagnostic analysis problems include four categories: time series correlation, comparison and screening of influencing factors, health diagnosis, and multivariate correlation; Predictive analysis problems include regression problems and alarm prediction; Decision analysis problems involve finding the optimal solution; Step 2: Design the analysis problem definition and convert the specific analysis problem into a JSON object; Step 3: Build a knowledge search module, generate corresponding search strategies based on the problem json object, use the knowledge graph as the search space to search for solutions that match the problem, and display and interact in a visual interface; Step 3.1, read and parse the specific analysis problem json object, and initialize the solution search control strategy; Step 3.2: Solve the analysis problem based on the search control strategy initialized in step 3.1 using the knowledge graph as the search space; Step 3.3, repeat step 3.2 until the current node attribute in the control strategy is the same as the target node attribute; Step 3.4: Screen the alternative solutions obtained in step 3.3 according to the functional requirements of the solution in the control strategy, and eliminate the solutions with incomplete functions; if there are multiple alternative solutions with complete functions, introduce the shortest path algorithm to further evaluate the alternative solutions, and select the solution with the shortest path and the lowest implementation cost as the search result; if there is no alternative solution with complete functions, select the solution with the most complete functions as the search result; Step 3.5, display the search results obtained in step 3.4 in the form of nodes and directed relationships in a visual interactive page; Step 3.6: Import the solution process finally determined in the visual interactive interface in step 3.5 in the form of a json object; Step 4: Build an automatic calculation module to automatically parse the solution into an executable program and display the analysis results based on the visualization component; Step 5: Solidify and restore the processing flow program. Solidify the processing flow program generated in step 4 into a json object file that complies with the PMML standard.

2. The method for automatically solving data analysis problems based on knowledge search according to claim 1, characterized in that: In step 2: For the rail equipment as the analysis object, the complete description of the analysis problem needs to include the analysis subject, problem description, known conditions, constraints, solution and expected results.

3. The method for automatically solving data analysis problems based on knowledge search according to claim 1 is characterized in that: In step 4: Step 4.1, read the solution json object finally generated in step 3, parse the processing nodes in the solution, and generate the corresponding processing component program; Import the json object of the processing node into the pre-set program template, automatically generate the corresponding processing code based on the statement splicing method, and compile it in the memory to generate the corresponding component program; Step 4.2: Assemble the component programs generated in step 4.1 into the computing engine pipeline in the form of a workflow according to the sequence of processing nodes in the solution; the last component in the pipeline acts as an estimator, and the other components work as converters, and the output result is the prediction result of the last component; Step 4.3: The output result is json format data that is compatible with the visualization component. Import the output result into the corresponding visualization component to directly generate a visualization analysis chart to display the data.

4. The method for automatically solving data analysis problems based on knowledge search according to claim 1, characterized in that: In step 5: Step 5.1: Solidify the processing flow program. Import the processing flow program in step 4 into the PMMLpipeline method of the sklearn library. Record the model structure, model parameters, and hyperparameters of the configuration model in the order of each node in the process based on the PMML standard, and solidify them in the form of a json file. Step 5.2: The processing flow program is restored. Based on the from_file method in the pypmml.Model library, the PMML standard file is read and parsed, and the processing flow program solidified in the file is reloaded into the memory for use.

Images