Data analysis system and method

Abstract

The invention provides a data analysis system and method, and the system comprises a front-end service and a rear-end service. The front-end service comprises a user interaction module, a data acquisition module and a visualization module; the front-end service comprises a high-performance computing module, the rear-end service comprises a high-performance computing module, the front-end service is designed and developed by a python language, and the rear-end service is designed and developed by a JAVA language; the data acquisition module is used for receiving the data analysis request and acquiring to-be-analyzed data; the high-performance calculation module is used for analyzing the to-be-analyzed data to obtain a target analysis result; the visualization module is used for carrying out visualization processing on the target analysis result; and the user interaction module is also used for displaying the visual analysis result to the user. In the system, the front-end service is designed and developed by the python language, the back-end service is designed and developed by the JAVA language, and the design of fusing the python language and the JAVA language is adopted, so that the respective characteristics of the python language and the JAVA language can be effectively exerted, and the performance of the data analysis system is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data analysis system and method. Background Art

[0002] Data analysis systems have irreplaceable value for financial business entities such as banks and investment institutions. They can not only improve business efficiency, reduce risks, and optimize resource allocation, but also meet users' diverse needs for real-time monitoring, analysis, and prediction of financial markets.

[0003] In the prior art, data analysis systems often use a single language design architecture. For example, if Python is used alone for design, although it has high development efficiency and low resource consumption, Python, as an interpreted language, may encounter performance bottlenecks when processing large amounts of data, and Python's multi-threading support is relatively insufficient. If Java is used alone for design, although it has better multi-threading support and higher performance, the amount of code is usually larger, the development efficiency is relatively low, and it is not as good as Python in drawing and processing data. It can be seen that the performance of existing data analysis systems is relatively low. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a data analysis system and method to improve the performance of the data analysis system.

[0005] To achieve the above objectives, the present application discloses the following technical solutions:

[0006] In a first aspect, the present application provides a data analysis system, comprising:

[0007] A data analysis system, characterized in that the system includes a front-end service and a back-end service; the front-end service includes a user interaction module, a data acquisition module and a visualization module; the back-end service includes a high-performance computing module, wherein the front-end service is designed and developed by Python language, and the back-end service is designed and developed by JAVA language;

[0008] The user interaction module is used to provide a user interaction display page;

[0009] A data collection module, used to receive a data analysis request sent by a user based on a user interaction page, and collect data to be analyzed based on the data analysis request;

[0010] A high performance computing module, used to analyze the data to be analyzed to obtain target analysis results;

[0011] The visualization module is used to perform visualization processing on the target analysis result to obtain a visualization analysis result;

[0012] The user interaction module is also used to display the visual analysis results to the user through the user interaction display page.

[0013] Optionally, the front-end service further includes: a preliminary analysis module, which is used to perform preliminary screening on the data to be analyzed according to the data analysis model to obtain preliminary screening data; the data analysis model includes: an improved isolation forest algorithm;

[0014] The high performance computing module is specifically used to perform data analysis on the preliminary screening data to obtain the target analysis results.

[0015] Optionally, the preliminary analysis module specifically includes:

[0016] A determination submodule is used to determine a plurality of dimensional information in the data to be analyzed and the priorities corresponding to the plurality of dimensional information.

[0017] A construction submodule, configured to select at least one of the plurality of dimensional information to construct an isolated tree according to the priority;

[0018] A cutting submodule is used to perform data cutting based on the isolated tree to obtain the preliminary screening data.

[0019] Optionally, the cutting submodule is specifically used to cyclically execute the following steps until the preliminary screening data is obtained:

[0020] Obtaining a data set of established nodes in the isolated tree;

[0021] Determining local distribution information of the data set and a segmentation result of the data set;

[0022] Dynamically adjust the cutting rule according to the local distribution information and the cutting result to obtain an adjusted cutting rule;

[0023] The data of the next node of the isolated tree is cut according to the adjusted cutting rule.

[0024] Optionally, the preliminary analysis module further includes:

[0025] A determination submodule is used to determine the time information in the data to be analyzed;

[0026] A windowing submodule, used for performing windowing processing on the data to be analyzed based on the time information to obtain the data to be analyzed in a first window and the data to be analyzed in a second window;

[0027] A preliminary cutting submodule, used for performing preliminary screening of the data to be analyzed in the first window according to the data analysis model to obtain first preliminary screening data of the data to be analyzed in the first window;

[0028] A model updating unit, configured to update the data analysis model according to the first preliminary screening data to obtain an updated data analysis model;

[0029] The preliminary cutting submodule is further used to perform preliminary screening on the data to be analyzed in the second window according to the updated data analysis model to obtain second preliminary screening data of the data to be analyzed in the second window.

[0030] Optionally, the high performance computing module is specifically used to divide the analysis of the data to be analyzed into processing tasks in multiple stages based on the pipeline model;

[0031] Based on the Spark computing engine, multiple processing tasks of the above stages are processed in parallel to obtain the target analysis results.

[0032] Optionally, the front-end service also includes a data cleaning module;

[0033] The data cleaning module is used to clean the data to be analyzed to obtain cleaned data to be analyzed, wherein the data cleaning includes one or more of the following: duplicate processing, missing value processing, abnormal value processing, and unified formatting;

[0034] The high performance computing module is specifically used to analyze the cleaned data to be analyzed to obtain the target analysis result.

[0035] In a second aspect, the present application provides a data analysis method, which is applied to the system as described in any one of the first aspects, and the method includes:

[0036] Provide user interactive display page;

[0037] Receiving a data analysis request sent by a user based on a user interaction page, and collecting data to be analyzed based on the data analysis request;

[0038] Analyzing the data to be analyzed to obtain target analysis results;

[0039] Performing visualization processing on the target analysis result to obtain a visualization analysis result;

[0040] The visual analysis results are presented to the user through the user interaction presentation page.

[0041] Optionally, before analyzing the data to be analyzed to obtain a target analysis result, the method further includes:

[0042] Preliminary screening of the data to be analyzed is performed according to the data analysis model to obtain preliminary screening data; the data analysis model includes: an improved isolation forest algorithm;

[0043] The step of analyzing the data to be analyzed to obtain a target analysis result includes: performing data analysis on the preliminary screening data to obtain the target analysis result.

[0044] Optionally, the method further includes:

[0045] Determine multiple dimensional information in the data to be analyzed and the priorities corresponding to the multiple dimensional information,

[0046] According to the priority, selecting at least one dimension information from the plurality of dimension information to construct an isolation tree;

[0047] Data cutting is performed based on the isolated tree to obtain the preliminary screening data.

[0048] The present application provides a data analysis system and method. In a data analysis system, the system includes a front-end service and a back-end service; the front-end service includes a user interaction module, a data acquisition module and a visualization module; the back-end service includes a high-performance computing module, wherein the front-end service is designed and developed by the python language, and the back-end service is designed and developed by the JAVA language; the user interaction module is used to provide a user interaction display page; the data acquisition module is used to receive a data analysis request sent by a user based on the user interaction page, and collect the data to be analyzed based on the data analysis request; the high-performance computing module is used to analyze the data to be analyzed and obtain the target analysis result; the visualization module is used to visualize the target analysis result and obtain the visualization analysis result; the user interaction module is also used to display the visualization analysis result to the user through the user interaction display page. In the above system, the front-end service is designed and developed by the python language, and the back-end service is designed and developed by the JAVA language. The design of the fusion of python and JAVA can effectively give play to the respective characteristics of python and JAVA and improve the performance of the data analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is a structural diagram of a data analysis system provided in an embodiment of the present application;

[0051] Figure 2 is a structural diagram of another data analysis system provided in an embodiment of the present application;

[0052] Figure 3 is a structural diagram of another data analysis system provided in an embodiment of the present application;

[0053] Figure 4 is an interactive schematic diagram of a data analysis method provided in an embodiment of the present application;

[0054] Figure 5 It is a flow chart of a data analysis method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] To facilitate understanding of the technical solution provided by this application, the background technology involved in this application will be described below.

[0057] In the prior art, data analysis systems often use a single language design architecture. For example, if Python is used alone for design, although it has high development efficiency and low resource consumption, Python, as an interpreted language, may encounter performance bottlenecks when processing large amounts of data, and Python's multi-threading support is relatively insufficient. If Java is used alone for design, although it has better multi-threading support and higher performance, the amount of code is usually larger, the development efficiency is relatively low, and it is not as good as Python in drawing and processing data. It can be seen that the performance of existing data analysis systems is relatively low.

[0058] The applicant also found that some specific problems may be more suitable for Python, such as data cleaning, visualization, etc., while other problems are more suitable for Java, such as high-performance computing, concurrent processing, etc.

[0059] Based on this, the present application provides a data analysis system and method. In a data analysis system, the system includes a front-end service and a back-end service; the front-end service includes a user interaction module, a data acquisition module and a visualization module; the back-end service includes a high-performance computing module, wherein the front-end service is designed and developed by the python language, and the back-end service is designed and developed by the JAVA language; the user interaction module is used to provide a user interaction display page; the data acquisition module is used to receive a data analysis request sent by a user based on the user interaction page, and collect the data to be analyzed based on the data analysis request; the high-performance computing module is used to analyze the data to be analyzed and obtain the target analysis result; the visualization module is used to visualize the target analysis result and obtain the visualization analysis result; the user interaction module is also used to display the visualization analysis result to the user through the user interaction display page. In this system, the front-end service is designed and developed by the python language, and the back-end service is designed and developed by the JAVA language. The design of the fusion of python and JAVA can effectively give play to the respective characteristics of python and JAVA and improve the performance of the data analysis system.

[0060] To facilitate understanding of the technical solution provided by the present application, the data analysis system provided by the present application will be described below in conjunction with the accompanying drawings. Figure 1 , which is a structural schematic diagram of a data analysis system provided in an embodiment of the present application, such as Figure 1 As shown, the system includes a front-end service and a back-end service; the front-end service includes a user interaction module 101 , a data acquisition module 102 , and a visualization module 103 . The back-end service includes a high-performance computing module 104 .

[0061] The user interaction module 101 is used to provide a user interaction display page.

[0062] In an embodiment of the present application, a user interaction display page can be designed and developed using the Python language, so that users can send data analysis requests, view data analysis results, etc. based on the user interaction page.

[0063] The data collection module 102 is used to receive a data analysis request sent by a user based on a user interaction page, and collect data to be analyzed based on the data analysis request.

[0064] In the embodiment of the present application, the present application does not limit the data content targeted by the data analysis request. As an example, the data analysis request may be a request for analyzing financial data, requesting to analyze whether there is abnormal data in the financial data. As another example, the data analysis request may also be a request for analyzing industrial data, requesting to analyze whether there is abnormal data in the industrial data.

[0065] In the embodiment of the present application, the user can directly upload the data to be analyzed based on the user interaction page, such as by uploading a file, thereby sending a data analysis request to the system, so that the data collection module 102 can collect the data to be analyzed in the file based on the data analysis request. As another example, the user can also upload the address information of the data source, so that the data to be analyzed in the data source can be collected through the address information of the data source.

[0066] The high performance computing module 104 is used to analyze the data to be analyzed to obtain target analysis results.

[0067] In the embodiment of the present application, the high-performance computing module developed by the JAVA language can better adapt to the processing of large amounts of data. The present application does not limit the specific content of the data to be analyzed. As an example, the present application can analyze the financial data to be analyzed, and the target analysis result is the abnormal transaction data in the financial data to be analyzed. As another example, the data analysis request can also be a request to analyze industrial data, and the target analysis result is the abnormal data existing in the industrial data.

[0068] The visualization module 103 is used to perform visualization processing on the target analysis result to obtain a visualization analysis result.

[0069] In the embodiment of the present application, after obtaining the target analysis result, the target analysis result can also be visualized through the visualization module of the front-end service. It can be understood that visualization refers to processing the target analysis result into a form of graphics and / or reports that are easy for users to understand. The visualization of the target analysis result can be better realized through the Python language, thereby improving the performance of visualization and reducing the difficulty of development.

[0070] The user interaction module 101 is also used to display the visual analysis results to the user through a user interaction display page.

[0071] In an embodiment of the present application, the visual analysis results will be displayed on the user interaction display page to better present the analysis results to the user.

[0072] In the above system, the front-end service is designed and developed in Python, and the back-end service is designed and developed in JAVA. The design of the fusion of Python and JAVA can effectively bring into play the respective characteristics of Python and JAVA and improve the performance and development efficiency of the data analysis system.

[0073] As a possible implementation, Figure 2 As shown, Figure 2A structural diagram of another data analysis system provided in an embodiment of the present application, wherein the front-end service also includes: a preliminary analysis module 105, which is used to perform preliminary screening of the data to be analyzed according to the data analysis model to obtain preliminary screened data; the data analysis model includes: an improved isolation forest algorithm.

[0074] In the embodiment of the present application, the front-end service also includes a preliminary analysis module 105, and the improved isolation forest algorithm in the data analysis model can also be used to perform preliminary screening of the data to be analyzed to obtain preliminary screening data. It can be understood that the preliminary screening data refers to the data after the preliminary screening of the data to be analyzed. For example, if it is necessary to analyze whether there is abnormal data in the financial data, the present application can use the preliminary analysis module 104 to screen out the preliminary screening data with a greater abnormal risk, and then use the high-performance computing module 104 to perform data analysis on the preliminary screening data to obtain the target analysis result.

[0075] As a possible implementation, the preliminary analysis module 105 specifically includes: determining submodules, constructing submodules, and cutting submodules.

[0076] The determination submodule is used to determine multiple dimensional information in the data to be analyzed and the priorities corresponding to the multiple dimensional information.

[0077] It can be understood that the data to be analyzed may include information of multiple dimensions. For example, for financial data, it may include dimensional information of multiple dimensions such as transaction time, transaction amount, transaction frequency, counterparty type, geographic location, etc. In an embodiment of the present application, the priorities corresponding to different dimensional information can be pre-set, so that when determining which dimensional information is included in the data to be analyzed, the priorities corresponding to each dimensional information can be directly determined.

[0078] This application does not limit the specific method of presetting the priorities corresponding to different dimensional information. As an example, this application can determine the priorities corresponding to different dimensional information according to the importance score of each dimensional information. The importance score can be determined based on the contribution of the dimensional information to the overall data variation.

[0079] The construction submodule is used to select at least one dimension information from the multiple dimension information to construct an isolation tree according to the priority.

[0080] In an embodiment of the present application, at least one dimension information may be selected to construct an isolation tree according to the priority of the dimension information. As an example, the present application may sort the priority of the dimension information and select the first one or more items in the priority sequence as the dimension information for constructing the isolation tree.

[0081] The cutting submodule is used to perform data cutting based on the isolated tree to obtain preliminary screening data.

[0082] In the embodiment of the present application, the process of establishing an isolation tree is the process of cutting data. It can be understood that data cutting is the process of screening data, and abnormal data can be cut out. The present application uses an improved isolation forest algorithm to flexibly adjust the dimensional information for cutting according to different data, and can select any number of dimensional information as cutting conditions according to priority, which can effectively improve the accuracy of data analysis.

[0083] The present application does not limit the specific method of data cutting based on the isolated tree. As an example, the cutting submodule is specifically used to cyclically execute the following steps A1-A4 until the preliminary screening data is obtained:

[0084] A1: Get the dataset of established nodes in the isolated tree.

[0085] In an embodiment of the present application, during the establishment of an isolated tree, the data to be analyzed can be cut. Each time the cut is made, the data in the current node can be cut into two parts, one is a data set whose dimensional information is greater than a threshold value P, and the other is a data set whose dimensional information is less than the threshold value P.

[0086] A2: Determine the local distribution information of the data set and the segmentation result of the data set.

[0087] In the embodiment of the present application, the data set of the established node can be statistically analyzed to determine the local distribution information. It can be understood that the local distribution information refers to the feature information of the data feature of the current node that is different from the overall data set, and the cutting result of the data set can be the information such as the accuracy rate of abnormal detection and the false alarm rate in the node. The local distribution information and the cutting result can be obtained according to the training related model, and the present application will not repeat them.

[0088] A3: Dynamically adjust the cutting rule according to the local distribution information and the cutting result to obtain an adjusted cutting rule.

[0089] It can be understood that the cutting rule includes a cutting threshold value P. The present application can flexibly adjust the cutting threshold value P according to the local distribution information and the cutting result.

[0090] A4: Perform data cutting of the next node according to the adjusted cutting rule.

[0091] In an embodiment of the present application, after obtaining the local distribution information and cutting results, they can be used as feedback conditions to form a closed-loop optimization, and the cutting rules can be flexibly adjusted to ensure that the cutting can maximize the separation of normal data and abnormal data and improve the accuracy of data analysis.

[0092] As a possible implementation method, the preliminary analysis module also includes:

[0093] The determination submodule is used to determine the time information in the data to be analyzed.

[0094] In the embodiment of the present application, when the data to be analyzed is real-time data, such as financial data, the present application can also determine the time information in the data to be analyzed. As an example, the time information can be transaction time, transaction interval, time trend, etc.

[0095] The windowing submodule is used to perform windowing processing on the data to be analyzed based on the time information to obtain the data to be analyzed in a first window and the data to be analyzed in a second window.

[0096] In an embodiment of the present application, the data to be analyzed can be windowed based on the time information in the data to be analyzed, that is, the data to be analyzed can be split according to the time series to obtain multiple windows of data to be analyzed. For example, the data to be analyzed can be split into the data to be analyzed of the first window and the data to be analyzed of the second window. In an embodiment of the present application, the time of the first window can be earlier than the time of the second window.

[0097] The preliminary cutting submodule is used to perform preliminary screening on the data to be analyzed in the first window according to the data analysis model to obtain first preliminary screening data of the data to be analyzed in the first window.

[0098] A model updating unit is used to update the data analysis model according to the first preliminary screening data to obtain the updated data analysis model.

[0099] In an embodiment of the present application, the data in each window can be regarded as an independent sample to be analyzed, so that after the data to be analyzed in the first window is preliminarily screened to obtain the first preliminary screening data, the data analysis model can be updated according to the first preliminary screening data to obtain an updated data analysis model. The present application does not limit the specific method of updating the data analysis model according to the first preliminary screening data. As an example, the present application can detect the anomaly detection accuracy of the first preliminary screening data, thereby adjusting the model parameters of the data analysis model according to the detection accuracy to obtain an updated data analysis model.

[0100] The preliminary cutting submodule is further used to perform preliminary screening on the data to be analyzed in the second window according to the updated data analysis model to obtain second preliminary screening data of the data to be analyzed in the second window.

[0101] In the embodiment of the present application, after obtaining the updated data analysis model, the data to be analyzed in the subsequent window can be analyzed according to the updated data analysis model. When the present application collects the data to be analyzed for data analysis each time, the data to be analyzed collected at that time can be split according to the time information, and the data analysis model can be updated during the process of analyzing the data to be analyzed, so as to further improve the accuracy of the data analysis.

[0102] As a possible implementation method, the high-performance computing module 104 in the present application is specifically used to divide the analysis of the data to be analyzed into multiple stages of processing tasks based on the pipeline model; and to parallelly process the processing tasks of multiple stages based on the Spark computing engine to obtain the target analysis results.

[0103] It can be understood that in the embodiments of the present application, the high-performance computing module can be combined with the pipeline model to decompose the analysis of the data to be analyzed into multiple stages, and use the Spark computing engine to parallelly process the tasks of multiple stages, thereby improving the efficiency and performance of the system.

[0104] Specifically, the present application can integrate the processing tasks of multiple stages into the Spark Streaming job through the Spark computing engine, implement the analysis process of the data to be analyzed, and improve the timeliness and accuracy of data processing.

[0105] As a possible implementation method, the front-end service also includes a data cleaning module;

[0106] The data cleaning module is used to clean the data to be analyzed to obtain cleaned data to be analyzed. The data cleaning includes one or more of the following: duplicate processing, missing value processing, abnormal value processing, and unified formatting.

[0107] It is understandable that duplicate processing refers to identifying and deleting duplicate information. As an example, when the data to be analyzed is financial data, the customer number and transaction time can be used as unique identifiers to identify and delete duplicate transaction records.

[0108] Missing value processing refers to supplementing the missing objects in the data to be analyzed. As an example, when the data to be analyzed is financial data, the missing transaction amount field is filled with the customer's recent average transaction amount; for the missing customer address field, since this field has little impact on fraud detection analysis, choose to delete it directly.

[0109] Outlier processing refers to marking or deleting data that is obviously abnormal. As an example, when the data to be analyzed is financial data, a box plot can be used to identify records with abnormally high transaction amounts. After analysis, it is confirmed that these are normal large-amount transfers by customers, so these records are retained but specially marked.

[0110] Data formatting refers to uniformly formatting the data to be analyzed. For example, when the data to be analyzed is financial data, the transaction date field can be uniformly formatted into the "yyyy-mm-dd" format, and the transaction amount field can be rounded to two decimal places.

[0111] The high performance computing module 104 is specifically used to analyze the cleaned data to be analyzed to obtain the target analysis result. The present application can effectively improve the accuracy of data analysis by cleaning the data and then further analyzing it.

[0112] See also Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of the structure of another data analysis system provided in the embodiment of the present application. Figure 3 As shown in the figure, the data analysis system adopts a microservice architecture, which divides the system into multiple independent service modules, and each module communicates with each other through RESTful API. The system mainly consists of Python front-end service, Java back-end service, database service and message queue service.

[0113] The Python front-end service includes a data acquisition module, a data cleaning module, a data analysis and visualization module, and a user interaction module. The data acquisition module is responsible for capturing data from financial data sources and supports timed or real-time acquisition. The data cleaning module cleans and formats the collected raw data. The data analysis and visualization module uses the data analysis library to conduct in-depth analysis of the data and provides visual reports. The user interaction module provides a Web interface that allows users to input data to be analyzed, view reports, and receive suggestions. It can be understood that the raw data is the data to be analyzed recorded above, and the data analysis and visualization module shown in the figure includes the preliminary analysis module and the visualization module recorded above.

[0114] The Java backend service includes a high-performance computing module, a RESTful API service module, a concurrent processing module, and a data storage and access module. The high-performance computing module executes complex financial models and algorithms to generate analysis results. The RESTful API service module provides an API interface for the front-end to call and implement data addition, deletion, modification, and query operations. The concurrent processing module handles a large number of concurrent requests to ensure that the system responds quickly and stably. The data storage and access module interacts with the database to implement persistent storage of data.

[0115] In addition, the role of the ZUUL gateway in the system is to limit and filter requests and unify the backend entrance. In addition, the system also includes database services and message queue services. The database service stores all the data required by the system, including user information, financial data, analysis results, etc. The message queue service is the middleware for front-end and back-end communication, realizing asynchronous data transmission and task scheduling. The financial model in the figure is a pipeline model, and the algorithm is a big data spark combined with machine learning algorithm.

[0116] In the above system, the front-end service is designed and developed in Python, and the back-end service is designed and developed in JAVA. The design of the fusion of Python and JAVA can effectively bring into play the respective characteristics of Python and JAVA and improve the performance of the data analysis system.

[0117] Figure 4 An interactive schematic diagram of a data analysis method provided in an embodiment of the present application, such as Figure 4 As shown in the figure, the data collection module of the Python front-end service collects raw data and passes it to the data cleaning module for cleaning. The cleaned data is passed to the high-performance computing module of the Java back-end service for in-depth analysis. After the Java back-end service completes the processing, the analysis results are returned to the Python front-end service through the RESTful API service module. The data analysis and visualization module of the Python front-end service receives the analysis results and performs visualization processing. The user interaction module displays the visualization results to the user, receives the data to be analyzed input by the user, and initiates the data analysis request again.

[0118] See also Figure 5 , Figure 5 This is a flow chart of a data analysis method provided in an embodiment of the present invention. The data analysis method provided in this application can be applied to the data analysis system provided in any of the above embodiments. The method includes: S101-S105.

[0119] S101: providing a user interaction display page;

[0120] S102: receiving a data analysis request sent by a user based on a user interaction page, and collecting data to be analyzed based on the data analysis request;

[0121] S103: Analyze the data to be analyzed to obtain target analysis results;

[0122] S104: Performing visualization processing on the target analysis result to obtain a visualization analysis result;

[0123] S105: Displaying the visualization analysis result to the user through the user interaction display page.

[0124] As a possible implementation manner, before S103 analyzes the data to be analyzed and obtains the target analysis result, the method further includes:

[0125] Preliminary screening of the data to be analyzed is performed according to the data analysis model to obtain preliminary screening data; the data analysis model includes: an improved isolation forest algorithm;

[0126] The step of analyzing the data to be analyzed to obtain a target analysis result includes: performing data analysis on the preliminary screening data to obtain the target analysis result.

[0127] As a possible implementation method, a data analysis method provided by the present application also includes:

[0128] Determine multiple dimensional information in the data to be analyzed and the priorities corresponding to the multiple dimensional information,

[0129] According to the priority, selecting at least one dimension information from the plurality of dimension information to construct an isolation tree;

[0130] Data cutting is performed based on the isolated tree to obtain the preliminary screening data.

[0131] As a possible implementation method, the data to be analyzed is preliminarily screened according to the data analysis model to obtain preliminary screening data, including:

[0132] The following steps are executed in a loop until the preliminary screening data is obtained: obtaining a data set of established nodes in the isolated tree; determining local distribution information of the data set and a cutting result of the data set; dynamically adjusting a cutting rule according to the local distribution information and the cutting result to obtain an adjusted cutting rule; and performing data cutting on the next node of the isolated tree according to the adjusted cutting rule.

[0133] As a possible implementation, the data to be analyzed is preliminarily screened according to the data analysis model to obtain the preliminary screening data, including the following steps:

[0134] Determine the time information in the data to be analyzed;

[0135] Performing window processing on the data to be analyzed based on the time information to obtain data to be analyzed in a first window and data to be analyzed in a second window;

[0136] Preliminarily screening the data to be analyzed in the first window according to the data analysis model to obtain first preliminary screening data of the data to be analyzed in the first window;

[0137] updating the data analysis model according to the first preliminary screening data to obtain an updated data analysis model;

[0138] The data to be analyzed in the second window is preliminarily screened according to the updated data analysis model to obtain second preliminarily screened data of the data to be analyzed in the second window.

[0139] As a possible implementation manner, S103 analyzes the data to be analyzed to obtain a target analysis result, including:

[0140] Dividing the analysis of the data to be analyzed into processing tasks in multiple stages based on the pipeline model;

[0141] Based on the Spark computing engine, multiple processing tasks of the above stages are processed in parallel to obtain the target analysis results.

[0142] As a possible implementation, the method further includes: performing data cleaning on the data to be analyzed to obtain cleaned data to be analyzed, wherein the data cleaning includes one or more of the following: duplicate processing, missing value processing, abnormal value processing, and unified formatting;

[0143] Then S103 analyzes the data to be analyzed to obtain a target analysis result, including: analyzing the cleaned data to be analyzed to obtain a target analysis result.

[0144] It should be noted that a data analysis method provided in an embodiment of the present application has the technical effect of any of the above embodiments, and the embodiments of the present application are not described in detail here.

[0145] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0146] It should be noted that a computer-readable storage medium provided in an embodiment of the present application has the technical effects of any of the above embodiments, and the embodiments of the present application are not described in detail here.

[0147] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0148] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0149] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data analysis system, characterized in that: The system includes a front-end service and a back-end service; the front-end service includes a user interaction module, a data acquisition module and a visualization module; the back-end service includes a high-performance computing module, wherein the front-end service is designed and developed by Python language, and the back-end service is designed and developed by JAVA language; The user interaction module is used to provide a user interaction display page; The data collection module is used to receive a data analysis request sent by a user based on the user interaction page, and collect data to be analyzed based on the data analysis request; The high performance computing module is used to analyze the data to be analyzed to obtain target analysis results; The visualization module is used to perform visualization processing on the target analysis result to obtain a visualization analysis result; The user interaction module is further used to display the visual analysis result to the user through the user interaction display page.

2. The system according to claim 1, characterized in that The front-end service also includes: a preliminary analysis module, which is used to perform preliminary screening on the data to be analyzed according to the data analysis model to obtain preliminary screening data; the data analysis model includes: an improved isolation forest algorithm; The high performance computing module is specifically used to perform data analysis on the preliminary screening data to obtain the target analysis results.

3. The system according to claim 2, characterized in that The preliminary analysis module specifically includes: A determination submodule is used to determine a plurality of dimensional information in the data to be analyzed and the priorities corresponding to the plurality of dimensional information. A construction submodule, configured to select at least one of the plurality of dimensional information to construct an isolated tree according to the priority; A cutting submodule is used to perform data cutting based on the isolated tree to obtain the preliminary screening data.

4. The system according to claim 3, characterized in that The cutting submodule is specifically used to cyclically execute the following steps until the preliminary screening data is obtained: Obtaining a data set of established nodes in the isolated tree; Determining local distribution information of the data set and a segmentation result of the data set; Dynamically adjust the cutting rule according to the local distribution information and the cutting result to obtain an adjusted cutting rule; The data of the next node of the isolated tree is cut according to the adjusted cutting rule.

5. The system according to claim 2, characterized in that The preliminary analysis module further includes: A determination submodule is used to determine the time information in the data to be analyzed; A windowing submodule, used for performing windowing processing on the data to be analyzed based on the time information to obtain the data to be analyzed in a first window and the data to be analyzed in a second window; A preliminary cutting submodule, used for performing preliminary screening of the data to be analyzed in the first window according to the data analysis model to obtain first preliminary screening data of the data to be analyzed in the first window; A model updating unit, configured to update the data analysis model according to the first preliminary screening data to obtain an updated data analysis model; The preliminary cutting submodule is further used to perform preliminary screening on the data to be analyzed in the second window according to the updated data analysis model to obtain second preliminary screening data of the data to be analyzed in the second window.

6. The system according to claim 1, characterized in that The high performance computing module is specifically used for: Dividing the analysis of the data to be analyzed into processing tasks in multiple stages based on the pipeline model; Based on the Spark computing engine, multiple processing tasks of the above stages are processed in parallel to obtain the target analysis results.

7. The system according to claim 1, characterized in that The front-end service also includes a data cleaning module; The data cleaning module is used to clean the data to be analyzed to obtain cleaned data to be analyzed, wherein the data cleaning includes one or more of the following: duplicate processing, missing value processing, abnormal value processing, and unified formatting; The high performance computing module is specifically used to analyze the cleaned data to be analyzed to obtain the target analysis result.

8. A data analysis method, the method being applied to the system according to any one of claims 1 to 7, the method comprising: Provide user interactive display page; Receiving a data analysis request sent by a user based on a user interaction page, and collecting data to be analyzed based on the data analysis request; Analyzing the data to be analyzed to obtain target analysis results; Performing visualization processing on the target analysis result to obtain a visualization analysis result; The visual analysis results are presented to the user through the user interaction presentation page.

9. The method according to claim 8, characterized in that Before analyzing the data to be analyzed to obtain the target analysis result, the method further includes: Preliminary screening of the data to be analyzed is performed according to the data analysis model to obtain preliminary screening data; the data analysis model includes: an improved isolation forest algorithm; The step of analyzing the data to be analyzed to obtain a target analysis result includes: performing data analysis on the preliminary screening data to obtain the target analysis result.

10. The method according to claim 9, characterized in that The method further comprises: Determine multiple dimensional information in the data to be analyzed and the priorities corresponding to the multiple dimensional information, According to the priority, selecting at least one dimension information from the plurality of dimension information to construct an isolation tree; Data cutting is performed based on the isolated tree to obtain the preliminary screening data.

Images