Data analysis method, system, device and storage medium

By connecting to multiple data sources and using Flink components for real-time data acquisition and correlation analysis, the problems of limited data monitoring range and slow processing speed in the existing technology are solved, efficient data analysis and display are achieved, and data analysis is improved The reliability and accuracy of the company provides strong data support for enterprise decision-making.

CN118484492BActive Publication Date: 2024-11-12宁波安得智联科技有限公司 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410950938.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-12
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Existing enterprise real-time data analysis methods cannot realize unified monitoring and correlation analysis of multiple data sources, resulting in limited data monitoring range, slow data processing speed and low reliability of data analysis results.

Method used

By identifying and connecting multiple data sources, using Flink components for real-time data acquisition and association analysis, combining with data warehouse clusters for long-term storage and backup, and displaying data processing results through visual devices, realizing dynamic configuration and user customization of data sources analyze.

Benefits of technology

It effectively improves the data monitoring range, data processing speed and reliability of data analysis results of data analysis, and can quickly and accurately process and display a large amount of data, providing data support for corporate decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118484492B_ABST
    Figure CN118484492B_ABST
Patent Text Reader

Abstract

The present application discloses a data analysis method, system, device and storage medium, and relates to the field of data processing technology. The data analysis method includes: identifying and accessing several data sources; based on a preset data collection method, using a data collection tool to collect real-time data from the several data sources to obtain data collection results; based on a preset data processing flow and a preset data processing mode, using a Flink component to perform data association analysis and processing on the data collection results to obtain data processing results; based on the data processing results and the preset data display style, displaying through a visualization device. Through the above scheme, the present application can not only effectively improve the data monitoring range, data processing speed and reliability of data analysis results of data analysis, but also can quickly and accurately process and display large amounts of data, providing data support for enterprise decision-making.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data analysis method, system, device and storage medium. Background Art

[0002] Real-time data processing refers to the process of collecting, processing and analyzing data instantly at the moment it is generated. Through real-time data processing, enterprises can obtain, process and apply data in a timely manner, so as to make decisions quickly, discover business opportunities and maximize the value of information.

[0003] Existing enterprise real-time data analysis methods can often only monitor specific data streams, and their data analysis results are not comprehensive enough, resulting in limited data monitoring scope and low reliability of data analysis results. In addition, the data processing speed of existing data analysis technologies is slow, and it is impossible to achieve real-time data monitoring and analysis at the second level.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data analysis method, system, device and storage medium, aiming to solve the technical problems of existing data analysis, such as limited data monitoring range, slow data processing speed and low reliability of data analysis results.

[0006] To achieve the above objectives, the present application proposes a data analysis method, which is applied to a data analysis system and includes:

[0007] Identify and access several data sources;

[0008] Based on a preset data collection method, using a data collection tool to collect real-time data from the plurality of data sources to obtain data collection results;

[0009] Based on the preset data processing flow and the preset data processing mode, the Flink component is used to perform data association analysis and processing on the data collection results to obtain data processing results;

[0010] Based on the data processing results and the preset data display style, the data is displayed through a visualization device.

[0011] In one embodiment, the step of performing data association analysis and processing on the data collection results based on the preset data processing flow and the preset data processing mode by using the Flink component to obtain the data processing results further includes:

[0012] Based on the data warehouse cluster, long-term storage and audit monitoring of the data processing results are performed; and / or

[0013] Based on a preset backup mechanism, the stored data in the data warehouse cluster is backed up to different locations regularly.

[0014] In one embodiment, before the step of performing real-time data collection on the plurality of data sources using a data collection tool based on a preset data collection method to obtain data collection results, the step further includes:

[0015] Receive function configuration parameters from the user;

[0016] Based on the function configuration parameters, a preconfigured data source, a preset data collection method, a preset data processing flow, a preset data processing mode, and a preset data display style are obtained by parsing;

[0017] Based on the preconfigured data source, several connected data sources are dynamically configured.

[0018] In one embodiment, the step of performing data association analysis and processing on the data collection results using the Flink component based on the preset data processing flow and the preset data processing mode to obtain the data processing results includes:

[0019] If the preset data processing mode is real-time stream processing, then according to the preset data processing flow, the Flink component is used to perform real-time data association analysis and processing on the data collection result to obtain a first data processing result;

[0020] If the preset data processing mode is batch processing, the Flink component is used to perform offline data association analysis and processing on the historical data collection results according to the preset data processing flow to obtain a second data processing result.

[0021] In one embodiment, before the step of performing real-time data association analysis and processing on the data collection results using the Flink component according to the preset data processing flow, the step further includes:

[0022] Based on a preset data quality control mechanism, the data collection result is quality checked, and data with abnormal data quality is filtered to obtain a first data collection result;

[0023] Based on a machine learning algorithm, clustering and predictive analysis are performed on the first data collection result to obtain a second data collection result;

[0024] The step of performing real-time data association analysis and processing on the data collection results using the Flink component according to the preset data processing flow includes:

[0025] According to the preset data processing flow, the Flink component is used to perform real-time data association analysis and processing on the second data collection result.

[0026] In one embodiment, after the step of performing real-time data association analysis and processing on the data collection results using the Flink component according to the preset data processing flow, the step further includes:

[0027] When a fault is detected in the real-time data association analysis and processing, the data collection results are re-analyzed and processed in real-time data association according to a preset retry mechanism through the displacement of the real-time data stream.

[0028] In one embodiment, the method further comprises:

[0029] Restrict data access rights for different users, establish a user access mechanism, and protect data security and privacy.

[0030] In addition, to achieve the above objectives, the present application also proposes a data analysis system, the system comprising:

[0031] A data source management module, used to identify and access several data sources;

[0032] A real-time data collection module is used to collect data from the plurality of data sources in real time using a data collection tool based on a preset data collection method to obtain data collection results;

[0033] A data processing module is used to perform data association analysis and processing on the data collection results using a Flink component based on a preset data processing flow and a preset data processing mode to obtain a data processing result;

[0034] A data storage module, used for long-term storage and management of the data processing results based on a data warehouse cluster; and / or regularly backing up the stored data in the data warehouse cluster to different locations based on a preset regular backup mechanism;

[0035] The presentation module is used to present the data through a visualization device based on the data processing results and a preset data presentation style.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a data analysis device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data analysis method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data analysis method described above are implemented.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] The data analysis method, system, device and storage medium proposed in the embodiments of the present application are specifically obtained by identifying and accessing several data sources; based on a preset data collection method, using a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; based on a preset data processing flow and a preset data processing mode, using a Flink component to perform data correlation analysis and processing on the data collection results to obtain data processing results; based on the data processing results and a preset data display style, displaying through a visualization device.

[0040] This application connects to multiple data sources, collects data from the data sources in real time, and uses Flink components to process and analyze the data collection results, thereby realizing correlation analysis of multiple data sources.

[0041] It can not only effectively improve the data monitoring scope, data processing speed and reliability of data analysis results, but also can quickly and accurately process and display large amounts of data, providing data support for corporate decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the data analysis method of this application;

[0045] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the data analysis method of this application;

[0046] Figure 3 A schematic diagram of a process flow provided for the third embodiment of the data analysis method of this application;

[0047] Figure 4A schematic diagram of a process flow provided for the fourth embodiment of the data analysis method of the present application;

[0048] Figure 5 A flowchart of the fifth embodiment of the data analysis method of the present application is provided;

[0049] Figure 6 A brief flowchart of the data analysis method provided in Example 1 of the present application;

[0050] Figure 7 This is a schematic diagram of the module structure of the data analysis system of the data analysis method embodiment of the present application;

[0051] Figure 8 The data analysis results for the storage site in the data analysis method embodiment of this application are involved;

[0052] Fig. 9 The data analysis results for storage equipment in the data analysis method embodiment of this application are involved;

[0053] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the data analysis method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: identify and access several data sources; based on a preset data collection method, use a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; based on a preset data processing flow and a preset data processing mode, use the Flink component to perform data correlation analysis and processing on the data collection results to obtain data processing results; based on the data processing results and a preset data display style, display them through a visualization device.

[0058] Technical terms involved in the embodiments of this application:

[0059] Apache Flink: Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed data flow engine written in Java and Scala. Flink is mainly used for stateful computing on bounded and unbounded data streams, supporting a variety of application scenarios such as real-time data processing, batch processing, event-driven applications, data lake computing, and machine learning. Flink is designed to run in all common cluster environments and achieve memory-level computing speeds at any scale.

[0060] Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner, and its pipeline runtime system can execute batch and stream processing programs. Flink provides a series of APIs (Application Programming Interface, application programming interface), including data stream API (for bounded or unbounded data streams), dataset API (for bounded datasets) and table API.

[0061] Apache Doris: Apache Doris is a high-performance, real-time analytical database based on the MPP architecture. It is well-known for its extremely fast and easy-to-use features. It can return query results under massive data with only a sub-second response time. It can not only support high-concurrency point query scenarios, but also support high-throughput complex analysis scenarios. Based on this, Apache Doris can better meet the use scenarios such as report analysis, instant query, unified data warehouse construction, data lake federated query acceleration, etc. Users can build user behavior analysis, AB experimental platform, log retrieval analysis, user portrait analysis, order analysis and other applications on it. Doris comes with an efficient columnar storage engine, which reduces the amount of data scanning while achieving an ultra-high data compression ratio, and supports multiple copies of data storage. The cluster has self-healing function, and its own distributed management framework can automatically manage the distribution, repair and balance of data copies. When the copy is damaged, the system can automatically sense and repair it.

[0062] Canal: The Canal component is an incremental subscription and consumption component based on the MySQL database binlog. Its core function is to provide incremental data subscription and consumption functions by parsing MySQL binlog log information.

[0063] Canal can detect changes in MySQL data d in real time by parsing MySQL binlog, and send the changed data to downstream consumers (such as Kafka, RocketMQ, etc.) or storage systems (such as Elasticsearch, HBase, etc.). Canal components are suitable for a variety of scenarios, including data synchronization, database backup, business decoupling, etc. Canal provides a variety of data interfaces, including database connectors, message queues, caches, etc., to help developers easily obtain and transmit data.

[0064] Since existing data analysis methods can only monitor specific data sources, they cannot achieve unified management and monitoring of multiple data sources, and they require a lot of time and computing resources, which cannot meet the needs of real-time monitoring and analysis. At the same time, traditional data analysis methods can only display the analysis results of a single data source, and cannot perform correlation analysis on multiple data sources, resulting in low reliability of data analysis results.

[0065] This application connects to multiple data sources, collects real-time data from the data sources, and uses Flink components to process and analyze the data collection results to achieve correlation analysis of multiple data sources. It can not only effectively improve the data monitoring scope, data processing speed and reliability of data analysis results, but also can quickly and accurately process and display large amounts of data, providing data support for corporate decision-making.

[0066] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a data analysis system capable of realizing the above functions, etc. The following takes the data analysis system as an example to illustrate this embodiment and the following embodiments.

[0067] Based on this, the present application embodiment provides a data analysis method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data analysis method of the present application.

[0068] In this embodiment, the method is applied to a data analysis system, and the method includes steps S110 to S140:

[0069] Step S110, identifying and accessing several data sources;

[0070] It should be noted that the data source is the source of data, which is a device or original media that provides certain required data. In this embodiment, the data source can be different types of databases, files, and application programming interfaces APIs.

[0071] Specifically, the data analysis system can identify different types of data sources, such as MySQL, Oracle, Hbase, SqlServer, Redis, MangoDB and other databases, and manage the connection of the data source, including the establishment, maintenance and closing of the connection, so as to achieve unified management of heterogeneous data sources. Heterogeneous data sources refer to multiple data sources with different data structures, access methods and forms.

[0072] It is understandable that in order to ensure the availability and stability of heterogeneous data sources, the data analysis system needs to monitor the status of several connected data sources, including the connection status of the data sources, performance indicators, etc.; at the same time, it is also necessary to perform dynamic testing on several connected data sources and reconnect when abnormal situations are found. After the reconnection fails, relevant personnel will be notified through SMS, voice, and APP message alarms for processing. Among them, the performance indicators of the data source refer to the response time of the data source, the number of user connections, the maximum number of concurrent users, the CPU or disk usage, the network bandwidth, etc.; dynamic testing refers to testing the stability and performance of the data source under real-time conditions. In this embodiment, in order to reduce the load on the system, the dynamic test can be set as a scheduled task.

[0073] Step S120, based on a preset data collection method, using a data collection tool to perform real-time data collection on the plurality of data sources to obtain data collection results;

[0074] It should be noted that the preset data collection method refers to the data collection method that can be supported by default in the data analysis system, including full collection, incremental collection, and incremental and full mixed collection.

[0075] Among them, full collection refers to extracting all data from the data source at one time during data synchronization or data extraction, regardless of whether the data has changed or not; incremental collection refers to the process of obtaining new data through different methods and technologies based on existing data, and incremental collection only extracts new or modified data; incremental full-volume hybrid collection refers to a strategy that combines full collection and incremental collection. When to use full collection and when to use incremental collection depends on factors such as the characteristics of the data source, the frequency of data updates, and business needs.

[0076] Specifically, according to the preset data collection method, the data collection tool is used to perform real-time data collection on the aforementioned connected data source, that is, to capture data changes in the connected data source in real time, including data addition, modification and deletion operations, and obtain data collection results.

[0077] Among them, data collection tools are tools used to obtain and collect data, including but not limited to web crawler tools, other professional data collection software, SQL Server Management Studio, MySQL Workbench and other database management tools.

[0078] In this embodiment, the data collection tool uses AnntoDCT, a data collection tool developed based on the open source component Canal. AnntoDCT supports full collection, incremental collection, and incremental and full mixed collection. AnntoDCT also supports the collection of multiple data sources, including mainstream relational databases such as MySQL and Oracle, and also supports non-relational databases such as HDFS, Redis, MangoDB, and Elasticsearch to provide flexible data collection solutions.

[0079] Furthermore, the data analysis system can store the data collection results in a local file system, such as HDFS (Hadoop Distributed File System), after compression and encryption. It also supports storage in message queues, achieving the scalability of data collection and outputting data collection results through multiple channels.

[0080] Among them, message queue refers to the use of efficient and reliable message transmission mechanism for data exchange. By providing message transmission and message queuing models, it can provide application decoupling, elastic scaling, redundant storage, traffic peak shaving, asynchronous communication, data synchronization and other functions in a distributed environment. Message queues include but are not limited to Apache Kafka, RabbitMQ, ApacheActiveMQ, RocketMQ, and Apache Pulsar.

[0081] Step S130, based on a preset data processing flow and a preset data processing mode, using the Flink component to perform data association analysis and processing on the data collection result to obtain a data processing result;

[0082] It should be noted that the preset data processing flow refers to the data processing flow that users customize based on their actual needs to meet the data processing needs in different business scenarios; the preset data processing mode refers to the data processing mode supported by the data analysis system by default, including real-time stream processing, batch processing, real-time stream and batch mixed processing and other processing modes.

[0083] Specifically, the Flink component is used to find the required data from the data collection results obtained above according to the preset data processing flow and preset data processing mode, and then the data is analyzed and processed in different dimensions, such as warehouse name, platform, business classification, industry, e-commerce category, customer and other dimensions, and finally the data processing results are obtained. Among them, the Flink component is a data processing component developed based on the open source Apache Flink framework.

[0084] In this embodiment, refer to Figure 8 , Fig. 9 As shown, users need to analyze warehouse data for the past week (03-20 to 03-26) according to their actual needs, including analysis of warehouse sites and equipment (such as forklifts). The preset data processing flow can be configured as a data analysis flow for calculating indicators such as site efficiency, site volume ratio, and equipment utilization.

[0085] Step S140: displaying the data through a visualization device based on the data processing result and a preset data display style.

[0086] It should be noted that the preset data display style refers to the data display style that the user customizes based on his or her actual needs to achieve user-customized data visualization result display.

[0087] Specifically, according to a preset data display style, the data processing results obtained above are pushed to a visualization device, and the data processing results are displayed in the form of computer graphics or images through the visualization device.

[0088] Furthermore, after the above step S130, steps A01-A02 are also included:

[0089] Step A01, based on the data warehouse cluster, long-term storage and audit monitoring of the data processing results; and / or

[0090] Step A02: Based on a preset backup mechanism, regularly back up the stored data in the data warehouse cluster to different locations.

[0091] It should be noted that a data warehouse cluster refers to a database cluster used to store data, and also refers to an overall system formed by combining multiple database instances (nodes); the goal of a database cluster is to improve the performance, availability, and scalability of a database system. In addition, it should be noted that a preset backup mechanism refers to data backup rules that relevant developers pre-set in a data analysis system based on actual data backup requirements.

[0092] It is understandable that in order to improve the reusability of data and reduce the load on the system, the data processing results after being processed by the Flink component can be stored and managed for a long time.

[0093] In this embodiment, the data warehouse cluster may be a high-performance data warehouse cluster AnntoDW that is secondary developed based on the open source Apache doris.

[0094] Specifically, based on the data warehouse cluster, the data processing results are persistently stored and audited, and the access records, modification records, deletion records, and update records of the data in the data warehouse cluster are monitored to promptly discover security issues such as unauthorized access, data leakage, data tampering, etc., thereby ensuring the reliability and security of the data stored in the data warehouse cluster.

[0095] Furthermore, in order to prevent data loss in the data warehouse cluster due to natural disasters or accidents, the data analysis system can back up and restore the data, regularly back up the data through a preset backup mechanism, and store the backup data in different locations, which can ensure that the data can be quickly restored in the event of a disaster or failure, thereby minimizing data loss and business impact.

[0096] The present embodiment provides a data analysis method, which specifically identifies and accesses several data sources; based on a preset data collection method, uses a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; based on a preset data processing flow and a preset data processing mode, uses a Flink component to perform data association analysis and processing on the data collection results to obtain data processing results; based on the data processing results and a preset data display style, displays them through a visualization device.

[0097] This application connects to multiple data sources, collects real-time data from the data sources, and uses Flink components to process and analyze the data collection results to achieve correlation analysis of multiple data sources. It can not only effectively improve the data monitoring scope, data processing speed and reliability of data analysis results, but also can quickly and accurately process and display large amounts of data, providing data support for corporate decision-making.

[0098] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , before the above step S120, it also includes steps S210 to S230:

[0099] Step S210, receiving function configuration parameters of the user;

[0100] It should be noted that functional configuration parameters refer to the parameters that users customize the configuration of various functions of the data analysis system based on their actual needs, including parameters for the connected data source, data collection method, data processing flow and mode, and data display method.

[0101] Specifically, the data analysis system provides an API interface to receive user-defined function configuration parameters through the API interface, so as to subsequently implement user-customized data processing and data display based on the function configuration parameters. It is understandable that the function configuration parameters provided by the API interface can also be displayed through a visualization device for users to customize.

[0102] Step S220, based on the function configuration parameters, parse to obtain a preconfigured data source, a preset data collection method, a preset data processing flow, a preset data processing mode, and a preset data display style;

[0103] Step S230: Dynamically configure the connected data sources based on the pre-configured data sources.

[0104] Specifically, the data analysis system parses the received functional configuration parameters to obtain the data source (i.e., preconfigured data source), preset data collection method, preset data processing flow, preset data processing mode, and preset data display style required by the user.

[0105] It should be understood that because the data source in Example 1 of the present application is automatically identified and accessed, there may be changes in the user's actual data analysis needs, which may result in the need to dynamically add, delete, or modify the data source to be configured.

[0106] Specifically, according to the preconfigured data sources, several data sources connected in the first embodiment of the present application are dynamically configured, that is, data sources are dynamically added, deleted or changed to meet the ever-changing needs of users.

[0107] The present embodiment provides a data analysis method, which specifically identifies and connects to several data sources; receives function configuration parameters of a user; based on the function configuration parameters, parses to obtain a preconfigured data source, a preset data collection method, a preset data processing flow, a preset data processing mode and a preset data display style; based on the preconfigured data source, dynamically configures the connected several data sources; based on the preset data collection method, uses a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; based on the preset data processing flow and the preset data processing mode, uses a Flink component to perform data association analysis and processing on the data collection results to obtain data processing results; based on the data processing results and the preset data display style, displays them through a visualization device.

[0108] Through the above scheme, this embodiment connects to multiple data sources, collects data from the data sources in real time, and uses Flink components to process and analyze the data collection results, thereby realizing correlation analysis of multiple data sources, which can effectively improve the data monitoring scope, data processing speed and reliability of data analysis results; at the same time, by receiving the user's function configuration parameters, user-customized data analysis and processing, as well as data analysis results display, can be performed to meet the data processing needs of different users.

[0109] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can refer to the above introduction, and will not be repeated later. Figure 3 , the step S130 includes steps S310 to S320:

[0110] Step S310: If the preset data processing mode is real-time stream processing, then according to the preset data processing flow, the Flink component is used to perform real-time data association analysis and processing on the data collection result to obtain a first data processing result;

[0111] In this embodiment, the Flink component supports real-time data processing and batch processing. Therefore, when the preset data processing mode is real-time stream processing, it is necessary to perform real-time data processing on the data collection results. At this time, the real-time requirements for data processing are often high, and the data processing time needs to reach the second level.

[0112] Then, the data analysis system will abstract the data collection results into several data streams according to the preset data processing flow customized by the user, and then divide the data streams into multiple data partitions. Then, based on the Flink component, multiple working nodes are used to process the data partitions in parallel to achieve parallel data processing; and the degree of correlation between variables in the data partitions is identified, and then the data partitions with data correlation are analyzed and processed to obtain the first data processing result.

[0113] Step S320: If the preset data processing mode is batch processing, the Flink component is used to perform offline data association analysis and processing on the historical data collection results according to the preset data processing flow to obtain a second data processing result.

[0114] Specifically, when the preset data processing mode is batch processing, it is necessary to perform offline data processing on the data collection results. At this time, data processing is performed on large batches of data, and the time consumption control for data processing is not strict. The batch processing of data can be planned to run when the data analysis system may be idle (such as at night) or during non-peak hours. It supports the configuration of alarm strategies for failed processing, such as alarms through voice, SMS, and APP messages.

[0115] The data analysis system will then collect the historical data collection results required by the user according to the user-defined preset data processing process, extract a large amount of data of some dimensions from it according to the user's actual needs, perform offline data association analysis and processing, and obtain the second data processing result.

[0116] The present embodiment provides a data analysis method, which specifically identifies and accesses several data sources; based on a preset data collection method, uses a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; if the preset data processing mode is real-time stream processing, then according to the preset data processing flow, uses the Flink component to perform real-time data association analysis and processing on the data collection results to obtain a first data processing result; if the preset data processing mode is batch processing, then according to the preset data processing flow, uses the Flink component to perform offline data association analysis and processing on historical data collection results to obtain a second data processing result; based on the data processing results and the preset data display style, the data is displayed through a visualization device.

[0117] Through the above solution, this embodiment connects to multiple data sources, collects data from the data sources in real time, and uses Flink components to perform real-time stream processing or batch processing on the data collection results based on user needs, thereby realizing correlation analysis of multiple data sources, which can effectively improve the data monitoring scope, data processing speed and reliability of data analysis results.

[0118] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above description, and will not be described in detail later. Figure 4 , the above step: before using the Flink component to perform real-time data association analysis and processing on the data collection results according to the preset data processing flow, it also includes:

[0119] Step S410, based on a preset data quality control mechanism, performing a quality check on the data collection result, and filtering data with abnormal data quality to obtain a first data collection result;

[0120] Step S420, performing clustering and prediction analysis on the first data collection result based on a machine learning algorithm to obtain a second data collection result;

[0121] It should be noted that the preset data quality control mechanism refers to the quality inspection rules set by relevant personnel based on experience or actual needs, as well as the response and processing strategy for abnormal quality data.

[0122] Specifically, according to the preset data quality control mechanism, the obtained data collection results are firstly checked for data quality, including data format, data outliers and missing values, data noise, etc. Then, the data with abnormalities in the data quality check are filtered to avoid the deviation of the data analysis results caused by data quality problems, and the data collection results after data filtering are obtained, that is, the first data collection results.

[0123] Then, based on machine learning algorithms, such as decision trees, random forests, support vector machines (SVM), naive Bayes, neural networks and other algorithms, the data features of the variables in the first data collection results are learned, including but not limited to time period features, other variable association features, etc., so as to deeply mine the first data collection results, discover hidden data association relationships, realize automatic data clustering, and predictive analysis of variables in the data, and save the clustering results, predictive analysis results and the first data collection results together to obtain the second data collection results.

[0124] Step S430, the step of using the Flink component to perform real-time data association analysis and processing on the data collection results according to the preset data processing flow includes:

[0125] According to the preset data processing flow, the Flink component is used to perform real-time data association analysis and processing on the second data collection result.

[0126] Specifically, the data analysis system uses the data collection results after data filtering and cluster prediction analysis as the real-time data stream to be processed according to the user-defined preset data processing flow, and uses multiple working nodes for parallel processing based on the Flink component; and identifies the degree of correlation between different cluster data in the real-time data stream, and then performs correlation data analysis and processing on the cluster data with data correlation.

[0127] Furthermore, in order to ensure the reliability and stability of the data analysis process, after the above step S310, step SB01 is also included:

[0128] Step SB01, when a fault is detected in the real-time data association analysis and processing, the data collection results are re-analyzed and processed in real time by using the displacement of the real-time data stream according to a preset retry mechanism.

[0129] It should be noted that the preset retry mechanism refers to the emergency measures pre-set by relevant developers in the event of data processing exceptions in the data analysis system. It supports customized settings based on actual user needs, such as limiting the number of retries, setting retry nodes, etc.

[0130] It is understandable that when data retry processing based on the preset retry mechanism fails, the data analysis system will notify relevant personnel for follow-up processing through SMS, voice and APP message alarms.

[0131] In addition, it should be noted that the displacement of the real-time data stream refers to the position information of the data that has been processed in the entire data stream during the real-time data processing.

[0132] Specifically, when the data analysis system detects the real-time stream processing mode, and a failure occurs in the system or a running node, resulting in an abnormality or failure in the real-time data association analysis and processing, the position of the processed data in the entire data stream is determined by the displacement of the real-time data stream, and then data processing is continued from the position of data that has not been processed based on the user-defined retry mechanism, without the need to reprocess the data, thereby ensuring continuous processing of the data stream and improving data processing efficiency.

[0133] The present embodiment provides a data analysis method, which specifically identifies and accesses several data sources; based on a preset data collection method, uses a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; if the preset data processing mode is real-time stream processing, then based on a preset data quality control mechanism, the data collection results are quality checked, and data with abnormal data quality are filtered to obtain a first data collection result; based on a machine learning algorithm, the first data collection results are clustered and predicted and analyzed to obtain a second data collection result; according to a preset data processing flow, the second data collection results are analyzed and processed in real time by using a Flink component to obtain a first data processing result; if the preset data processing mode is batch processing, then according to the preset data processing flow, the historical data collection results are analyzed and processed by using a Flink component to obtain an offline data association analysis and processing, and a second data processing result is obtained; based on the data processing results and a preset data display style, the data is displayed through a visualization device.

[0134] Through the above scheme, this embodiment performs quality inspection and filtering, as well as automatic clustering and predictive analysis on real-time data streams through a preset data quality control mechanism and machine learning algorithm, which not only ensures the data quality of real-time data streams, but also improves the accuracy and precision of data processing.

[0135] Based on the first embodiment and / or the second embodiment and / or the third embodiment and / or the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the above-mentioned embodiments 1, 2, 3 and 4 can be referred to the above introduction and will not be described in detail later. Figure 5 , the data analysis method further comprises step S510:

[0136] Step S510, restricting data access rights of different users, establishing a user access mechanism, and protecting data security and privacy.

[0137] Specifically, in this embodiment, facing users from different departments or different permission group levels within an enterprise or other organization, in order to prevent the leakage of critical or core data, it is necessary to restrict the data access rights of different users, that is, to restrict access to the data stored in the data warehouse cluster and data processing operations, including operations to add, delete, query and modify data.

[0138] This embodiment provides a data analysis method, which specifically identifies and accesses several data sources; based on a preset data collection method, uses a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; based on a preset data processing flow and a preset data processing mode, uses a Flink component to perform data association analysis and processing on the data collection results to obtain data processing results; based on the data processing results and a preset data display style, displays them through a visualization device; restricts data access rights for different users, establishes a user access mechanism, and protects data security and privacy.

[0139] Through the above scheme, this embodiment establishes a user access mechanism to perform hierarchical data permission access control on different users, prevent data leakage, achieve data security protection, and improve the security and privacy of internal data of an enterprise or organization.

[0140] For example, to help understand the implementation process of the data analysis method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 6 , Figure 6 A brief flow chart of a data analysis method is provided, specifically:

[0141] First, the data analysis method is applied to the real-time dashboard platform (i.e., the data analysis system). The real-time dashboard platform first connects to the heterogeneous data sources consisting of the MySQL database and the Oracle database, and then performs real-time data collection on the above heterogeneous data sources based on the self-developed data collection tool AnntoDCT, and pushes the data collection results to the message queue.

[0142] Then, in the data processing stage, the Flink real-time stream processing component is used to obtain the data collection results to be processed from the message queue, and data correlation analysis and processing are performed. The results obtained after data processing are then stored in the self-developed data warehouse cluster AnntoDW for storage and audit management. At the same time, the results obtained after data processing can also be pushed to other physical devices using the large-screen system of the presentation layer, and the data analysis results can be displayed through visual physical devices such as large-screen terminals, PC terminals, and mobile terminals.

[0143] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data analysis method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0144] This application also provides a data analysis system, please refer to Figure 7 , the data analysis system comprises:

[0145] A data source management module 10, used to identify and access a number of data sources;

[0146] The real-time data collection module 20 is used to collect data from the plurality of data sources in real time using a data collection tool based on a preset data collection method to obtain data collection results;

[0147] The data processing module 30 is used to perform data association analysis and processing on the data collection results by using the Flink component based on a preset data processing flow and a preset data processing mode to obtain a data processing result;

[0148] The data storage module 40 is used to perform long-term storage and management of the data processing results based on the data warehouse cluster; and / or to regularly back up the stored data in the data warehouse cluster to different locations based on a preset regular backup mechanism;

[0149] The presentation module 50 is used to present the data through a visualization device based on the data processing result and a preset data presentation style.

[0150] The data analysis system provided by the present application adopts the data analysis method in the above embodiment, which can solve the technical problems of the existing data analysis, such as limited data monitoring range, slow data processing speed and low reliability of data analysis results. Compared with the prior art, the beneficial effects of the data analysis system provided by the present application are the same as the beneficial effects of the data analysis method provided by the above embodiment, and the other technical features in the data analysis are the same as the features disclosed in the above embodiment method, which will not be described in detail here.

[0151] The present application provides a data analysis device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data analysis method in the above-mentioned embodiment one.

[0152] Reference below Fig.10 , which shows a schematic diagram of the structure of a data analysis device suitable for implementing an embodiment of the present application. The data analysis device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10The data analysis device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] like Fig.10 As shown, the data analysis device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data analysis device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data analysis device to communicate with other devices wirelessly or by wire to exchange data. Although the data analysis device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0154] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0155] The data analysis device provided by the present application adopts the data analysis method in the above embodiment, which can solve the technical problems of the existing data analysis, such as limited data monitoring range, slow data processing speed and low reliability of data analysis results. Compared with the prior art, the beneficial effects of the data analysis device provided by the present application are the same as the beneficial effects of the data analysis method provided by the above embodiment, and the other technical features in the data analysis device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0156] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0158] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data analysis method in the above-mentioned embodiment.

[0159] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0160] The computer-readable storage medium may be included in the data analysis device; or may exist independently without being assembled into the data analysis device.

[0161] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data analysis device, the data analysis device enables the data analysis device to: identify and access several data sources; based on a preset data collection method, use a data collection tool to perform real-time data collection on the several data sources to obtain data collection results; based on a preset data processing flow and a preset data processing mode, use a Flink component to perform data association analysis and processing on the data collection results to obtain data processing results; based on the data processing results and a preset data display style, display them through a visualization device.

[0162] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0165] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data analysis method, and can solve the technical problems of limited data monitoring range, slow data processing speed, and low reliability of data analysis results in existing data analysis. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data analysis method provided in the above-mentioned embodiment, and will not be elaborated here.

[0166] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A data analysis method, characterized in that: The method is applied to a data analysis system, and the method comprises: Identify and access several data sources; Based on a preset data collection method, a data collection tool is used to collect real-time data from the plurality of data sources to obtain data collection results, wherein the preset data collection method includes full collection, incremental collection, and incremental and full mixed collection, and the data collection tool is constructed based on the open source component Canal and supports data collection from multiple data sources; Based on the preset data processing flow and the preset data processing mode, the Flink component is used to perform data association analysis and processing on the data collection results to obtain data processing results; Based on the data processing results and the preset data display style, display is performed through a visualization device; The step of performing data association analysis and processing on the data collection results by using the Flink component based on the preset data processing flow and the preset data processing mode to obtain the data processing results includes: If the preset data processing mode is real-time stream processing, then according to the preset data processing flow, the Flink component is used to perform real-time data association analysis and processing on the data collection result to obtain a first data processing result; If the preset data processing mode is batch processing, the Flink component is used to perform offline data association analysis and processing on the historical data collection results according to the preset data processing flow to obtain a second data processing result; Before the step of performing real-time data association analysis and processing on the data collection results using the Flink component according to the preset data processing flow, the step further includes: Based on a preset data quality control mechanism, the data collection result is quality checked, and data with abnormal data quality is filtered to obtain a first data collection result; Based on a machine learning algorithm, learning the data association relationship in the first data collection result, and performing clustering and predictive analysis on the first data collection result according to the data association relationship to obtain a second data collection result; The step of performing real-time data association analysis and processing on the data collection results using the Flink component according to the preset data processing flow includes: According to a preset data processing flow, the Flink component is used to determine the degree of association between different clustered data in the second data collection result, and real-time data association analysis and processing are performed based on the degree of association; Before the step of performing real-time data collection on the plurality of data sources using a data collection tool based on a preset data collection method to obtain data collection results, the step further includes: Receive function configuration parameters from the user; Based on the function configuration parameters, a preconfigured data source, a preset data collection method, a preset data processing flow, a preset data processing mode, and a preset data display style are obtained by parsing; Based on the preconfigured data source, several connected data sources are dynamically configured.

2. The method according to claim 1, characterized in that After the step of performing data association analysis and processing on the data collection results based on the preset data processing flow and the preset data processing mode by using the Flink component to obtain the data processing results, the step further includes: Based on the data warehouse cluster, long-term storage and audit monitoring of the data processing results are performed; and / or Based on a preset backup mechanism, the stored data in the data warehouse cluster is backed up to different locations regularly.

3. The method according to claim 1, characterized in that After the step of performing real-time data association analysis and processing on the data collection results using the Flink component according to the preset data processing flow, the method further includes: When a fault is detected in the real-time data association analysis and processing, the data collection results are re-analyzed and processed in real-time data association according to a preset retry mechanism through the displacement of the real-time data stream.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Restrict data access rights for different users, establish a user access mechanism, and protect data security and privacy.

5. A data analysis system, characterized in that: The system comprises: A data source management module, used to identify and access several data sources; A real-time data collection module, which is used to collect real-time data from the plurality of data sources using a data collection tool based on a preset data collection method to obtain data collection results, wherein the preset data collection method includes full collection, incremental collection, and incremental full mixed collection, and the data collection tool is constructed based on the open source component Canal and supports data collection from multiple data sources; A data processing module is used to perform data association analysis and processing on the data collection results using a Flink component based on a preset data processing flow and a preset data processing mode to obtain a data processing result; A data storage module, used for long-term storage and management of the data processing results based on a data warehouse cluster; and / or regularly backing up the stored data in the data warehouse cluster to different locations based on a preset regular backup mechanism; A presentation module, used for presenting the data through a visualization device based on the data processing result and a preset data presentation style; The data processing module is further configured to, if the preset data processing mode is real-time stream processing, use the Flink component to perform real-time data association analysis and processing on the data collection results according to the preset data processing flow to obtain a first data processing result; if the preset data processing mode is batch processing, use the Flink component to perform offline data association analysis and processing on the historical data collection results according to the preset data processing flow to obtain a second data processing result; The data processing module is further used to perform a quality check on the data collection result based on a preset data quality control mechanism, and filter data with abnormal data quality to obtain a first data collection result; based on a machine learning algorithm, learn the data association relationship in the first data collection result, and perform clustering and predictive analysis on the first data collection result according to the data association relationship to obtain a second data collection result; according to a preset data processing flow, use the Flink component to determine the degree of association between different clustered data in the second data collection result, and perform real-time data association analysis and processing based on the degree of association; The data source management module is also used to receive the user's function configuration parameters; based on the function configuration parameters, parse to obtain the pre-configured data source, the preset data collection method, the preset data processing flow, the preset data processing mode and the preset data display style; based on the pre-configured data source, dynamically configure the connected data sources.

6. A data analysis device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data analysis method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data analysis method according to any one of claims 1 to 4 are implemented.