Real-time process mining method based on data analysis

Abstract

The invention discloses a real-time process mining method based on data analysis, which belongs to the technical field of data analysis, and comprises the following steps: providing a user interface, allowing a user to configure input parameters of a data acquisition task, including a target website URL (Uniform Resource Locator) and an acquisition frequency; the data acquisition module is used for realizing pre-definition and comprises a webpage crawler module which is used for automatically accessing and capturing contents of a target website and pre-processing the contents; an API calling module is provided and used for obtaining data from a third-party service or a data provider; the database query supporting module is used for directly extracting data from a database and carrying out encryption transmission on the data; monitoring the state and performance of the data acquisition task in real time; when abnormity or performance reduction is detected, the system automatically triggers an alarm; according to the method, the technical threshold of process mining is remarkably reduced through a user-friendly interface.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a real-time process mining method based on data analysis. Background Art

[0002] With the development of information technology, process mining has become an important means for enterprises and organizations to obtain information, analyze trends and make decisions. However, traditional process mining methods often require professional technicians to program and configure, which is not only time-consuming and labor-intensive, but also difficult to adapt to rapidly changing business needs. Therefore, how to provide a simple, easy-to-use, flexible and efficient data collection method has become an urgent problem to be solved.

[0003] The emergence of low-code platforms provides a new approach to solving this problem. Through visual interfaces and predefined modules, low-code platforms enable non-technical personnel to easily create and configure data collection tasks, greatly reducing the threshold and cost of process mining. At the same time, low-code platforms also support custom development and integration to meet the personalized needs of different users. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time process mining method based on data analysis to solve the following technical problems:

[0005] Traditional data collection methods often require professional technicians to program and configure, which is not only time-consuming and labor-intensive, but also difficult to adapt to rapidly changing business needs.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A real-time process mining method based on data analysis includes the following steps:

[0008] Provide a user interface that allows users to configure the input parameters of data collection tasks, including the target website URL and collection frequency;

[0009] Implement predefined data collection, including web crawler module, for automatically accessing and crawling the content of target websites and preprocessing them;

[0010] Provide API calling interface for obtaining data from third-party services or data providers;

[0011] Based on database query instructions, it is used to directly extract data from the database and encrypt the data for transmission.

[0012] As a further solution of the present invention: the pretreatment process is:

[0013] Automatically identify and remove duplicate data, format errors, and unstructured text data; apply natural language processing technology to extract key information, including entity recognition and sentiment analysis.

[0014] As a further solution of the present invention: the encrypted transmission includes using SSL / TLS protocol encryption during data transmission, and AES or RSA encryption when storing data; and implementing access control policies to authorize pre-selected users to access specific data.

[0015] As a further solution of the present invention: automatically create data collection tasks according to the data model specified by the user, including determining the collection points and data fields; automatically configure the execution order and frequency of the data collection tasks, and update the data regularly or in real time; provide a visual interface to display the data model and data collection tasks, so that the user can adjust and optimize the collection process.

[0016] As a further solution of the present invention: providing development tools and APIs to support users in developing and debugging custom data acquisition modules; allowing users to seamlessly integrate custom modules with existing data acquisition processes.

[0017] As a further solution of the present invention: using window functions and grouping operations, analyzing the browsing events and corresponding click events triggered by a specific user within a certain day; calculating the time difference between the browsing events and the click events, and determining the specific position of the user in the funnel conversion path;

[0018] The calculation results are further aggregated to count the funnel conversion rate of a specific user group within a certain day, including the number of conversions between different states; finally, the results are grouped by event date, remaining date, and date difference, and the total number of times in each group is calculated.

[0019] As a further solution of the present invention: analyzing the execution status and performance indicators of historical data collection tasks, including response time and data accuracy; and automatically adjusting the parameters of the data collection tasks based on the analysis results to increase or decrease the collection frequency.

[0020] As a further solution of the present invention: record and analyze the execution time, response speed and data accuracy of each data collection task, and define key performance indicators, including the average response time of data collection tasks, daily data update delays and abnormal task occurrence rate;

[0021] Use the report generation function provided by the low-code platform to regularly generate monitoring reports on the execution of historical data collection tasks, showing the trends and changes of key indicators;

[0022] Automatically detect and alert anomalies in data collection tasks, including data loss, field errors, or frequent execution errors, through preset anomaly detection algorithms or user-defined rules.

[0023] As a further solution of the present invention: a central monitoring dashboard is provided for real-time monitoring of the status and performance of data acquisition tasks; when an abnormality or performance degradation is detected, the system automatically triggers an alarm and recommends optimization measures; and a fault diagnosis tool is further provided.

[0024] Beneficial effects of the present invention:

[0025] The process mining based on the low-code platform provided by the present invention significantly reduces the technical threshold for data collection through a user-friendly interface, so that non-professionals can easily configure and manage collection tasks across multiple data sources; its multifunctional design integrates automated data preprocessing, security measures such as encrypted transmission and access control, and powerful data analysis and performance monitoring tools to ensure the security and accuracy of data collection; in addition, it supports automated adjustment and optimization of collection tasks, provides anomaly detection and alarms, and allows users to customize and expand data collection modules through development tools and APIs to meet personalized needs; the present invention improves the efficiency, quality and maintainability of data collection, and provides a flexible, easy-to-use and secure data collection solution for enterprises or organizations that require efficient data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below in conjunction with the accompanying drawings.

[0027] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 As shown, the present invention is a real-time process mining method based on data analysis, comprising the following steps:

[0030] 1. User interface design:

[0031] Input parameter configuration: Users can input the URL of the target website and the frequency of collection through the graphical user interface (GUI). The interface should provide friendly prompts and verification mechanisms to ensure the validity of user input.

[0032] Task management: Users can view the list of configured data collection tasks, including the status, last run time, and next run time of each task. Users can also edit or delete existing tasks.

[0033] Log viewing: Provides a log viewing function, users can monitor the execution of data collection tasks in real time, including the number of successfully captured data, the number of failures and error messages.

[0034] Implementation details:

[0035] Develop the user interface using a front-end framework such as React or Vue.js to ensure the interface is responsive and provides a good user experience.

[0036] The backend communicates with the frontend using a RESTful API, processes user input and stores task configurations.

[0037] The database is used to store task configuration and log information. It is recommended to use a relational database such as MySQL or PostgreSQL.

[0038] 2. Data Collection

[0039] Web crawler module:

[0040] Automatic access and crawling: Use Python's requests library to send HTTP requests to obtain the content of the target website. For dynamically loaded content, you can use tools such as Selenium or Playwright to simulate browser behavior.

[0041] Content preprocessing: The captured HTML content is parsed using libraries such as BeautifulSoup or lxml to extract the required data fields. The data is cleaned to remove useless information and duplicate data.

[0042] Data storage: Store the preprocessed data in a local file or directly in a database for subsequent processing and use.

[0043] 4. API call:

[0044] Third-party service integration: Integrate various third-party API services according to business needs, such as weather data, financial data, etc. Use Python's requests library or other HTTP client libraries to make API calls.

[0045] Error handling: Implement a robust error handling mechanism to ensure that retries or error logs are recorded when API calls fail.

[0046] Data caching: For frequently called third-party APIs, a data caching mechanism can be implemented to reduce unnecessary network requests and improve system performance.

[0047] 5. Database Query

[0048] Extract data directly from the database:

[0049] Database connection: Use the appropriate database driver (such as psycopg2 for PostgreSQL, pymysql for MySQL) to establish a connection to the database.

[0050] Query execution: Write SQL query statements according to user needs to extract required data from the database. Support complex query operations such as joins, groupings, and aggregations.

[0051] Data encryption transmission: During data transmission, the SSL / TLS protocol is used to encrypt the data to ensure data security. Python's cryptography library can be used to implement data encryption and decryption.

[0052] Implementation details:

[0053] The database query results can be further processed, such as formatting, filtering, and sorting, to meet the final application requirements.

[0054] In order to improve the scalability and maintainability of the system, it is recommended to encapsulate database operations into independent modules or classes for easy reuse and maintenance.

[0055] The present invention allows non-technical users to set data collection tasks by configuring input parameters by providing an intuitive user interface, thereby lowering the technical threshold for data collection and enabling a wider range of user groups to easily create and manage data collection processes.

[0056] The present invention integrates multiple data collection capabilities, including web crawlers, API calls, and database queries, making it a comprehensive solution that can cover multiple data sources and collection needs.

[0057] Automatically clean data, such as identifying and removing duplicate data, format errors and unstructured text data, and applying natural language processing technology to extract key information to improve data quality and analysis efficiency.

[0058] Use SSL / TLS protocol and AES or RSA encryption during data transmission and storage to ensure data security. At the same time, implement access control policies to limit data access rights and increase the level of data protection.

[0059] It supports automatic creation and configuration of data collection tasks based on user-specified data models, provides development tools and APIs to develop and integrate custom data collection modules, and implements personalized data collection solutions.

[0060] By analyzing the performance indicators of data collection tasks, such as response time and data accuracy, the system can automatically adjust the collection frequency, record and analyze execution time, response speed, etc., to improve the efficiency and accuracy of data collection.

[0061] Provides a dashboard for real-time monitoring of data collection status and performance, and is equipped with anomaly detection algorithms or rules to automatically detect and alarm abnormal situations such as data loss or frequent errors, ensuring the stability of data collection.

[0062] It allows complex analysis of collected data, such as window function operations and grouping, calculation of conversion and funnel conversion rates between browsing and click events, etc., to help better understand user behavior and optimize business decisions.

[0063] In another preferred embodiment of the present invention, the pretreatment process is:

[0064] Automatically identify and remove duplicate data, format errors, and unstructured text data; apply natural language processing technology to extract key information, including entity recognition and sentiment analysis.

[0065] In another preferred embodiment of the present invention, the encrypted transmission includes using SSL / TLS protocol encryption during data transmission, and AES or RSA encryption during data storage; and implementing access control policies to authorize pre-selected users to access specific data.

[0066] In another preferred embodiment of the present invention,

[0067] 1. Data model definition:

[0068] The user first needs to define a data model, which describes the structure, type, and relationship of the data to be collected. For example, for the collection of product information on an e-commerce website, the data model may include fields such as product name, price, inventory, and description.

[0069] The system provides a graphical interface or template to help users easily define data models without writing complex codes.

[0070] 2. Determination of collection points:

[0071] Based on the user-defined data model, the system automatically identifies and determines the key points of data collection. These collection points may be specific elements on a web page, API interfaces, or tables and columns in a database.

[0072] The system uses machine learning algorithms to analyze web page structure or API documentation and intelligently recommend the best collection point locations.

[0073] 3. Data field mapping:

[0074] The system automatically maps the fields in the data model to the collection points, ensuring that each field can obtain data from the correct location.

[0075] Users can review and adjust mapping results through the visual interface to ensure the accuracy of data collection.

[0076] 4. Execution order arrangement:

[0077] The system automatically generates the optimal data collection execution sequence based on the data model and the characteristics of the collection points. This helps to reduce collection time and improve efficiency.

[0078] For complex data collection tasks, the system supports parallel processing to further speed up the collection process.

[0079] 5. Frequency setting:

[0080] Users can set the frequency of data collection according to their needs, such as hourly, daily or weekly. The system will automatically schedule data collection tasks according to the set frequency.

[0081] For scenarios with high real-time requirements, the system supports real-time data collection to ensure the timeliness of the data.

[0082] 6. Data model display:

[0083] The system provides an intuitive visual interface to display the user-defined data model and its fields. Users can easily adjust the model structure by dragging and dropping.

[0084] The interface will also display the status, progress and history of the data collection task to facilitate user monitoring and management.

[0085] 7. Collection task display and adjustment:

[0086] The system displays all configured data collection tasks in a graphical manner, including task name, target website / API, collection frequency and other information.

[0087] Users can adjust the execution order, frequency and other parameters of tasks directly on the interface, and the system will update and apply these changes in real time.

[0088] 8. Collection process optimization:

[0089] The system has a built-in intelligent optimization algorithm that automatically adjusts the collection strategy to improve overall efficiency based on historical collection data and performance indicators.

[0090] Users can also manually set optimization rules, such as skipping data that has already been collected, prioritizing the collection of important fields, etc.

[0091] In another preferred embodiment of the present invention, development tools and APIs are provided to support users in developing and debugging custom data acquisition modules; and to allow users to seamlessly integrate custom modules with existing data acquisition processes.

[0092] In another preferred embodiment of the present invention, window functions and grouping operations are used to analyze browsing events and corresponding click events triggered by a specific user within a certain day; the time difference between the browsing event and the click event is calculated, and the specific position of the user in the funnel conversion path is determined;

[0093] The calculation results are further aggregated to count the funnel conversion rate of a specific user group within a certain day, including the number of conversions between different states; finally, the results are grouped by event date, remaining date, and date difference, and the total number of times in each group is calculated.

[0094] In another preferred embodiment of the present invention, the execution status and performance indicators of the historical data collection task are analyzed, including response time and data accuracy; and the parameters of the data collection task are automatically adjusted based on the analysis results to increase or decrease the collection frequency.

[0095] In another preferred embodiment of the present invention:

[0096] 1. Average response time:

[0097] The system records the time required for each data collection task from start to completion and calculates the average response time of all tasks. This indicator reflects the overall efficiency and stability of the system.

[0098] The average response time can be derived through statistical analysis, such as using the mean, median or 95% confidence interval.

[0099] 2. Daily data update delay:

[0100] The system monitors the difference between the actual completion time and the scheduled completion time of each data collection task, that is, the data update delay. This indicator helps to identify whether there is a data processing bottleneck or external factors affecting the data update speed.

[0101] Delays can be quantified by calculating the difference between the actual completion time and the scheduled completion time, and the daily delay trend can be displayed in a graph.

[0102] 3. Abnormal task occurrence rate:

[0103] The system counts the proportion of abnormal data collection tasks to the total number of tasks within a certain period of time, that is, the abnormal task occurrence rate. This indicator reflects the reliability and robustness of the system.

[0104] Abnormal tasks may include data loss, field errors, frequent execution errors, etc. The system will automatically identify these abnormal situations according to predefined rules and record relevant logs for subsequent analysis.

[0105] 4. Report generation function:

[0106] The system uses the report generation tool provided by the low-code platform to regularly generate monitoring reports on the execution of historical data collection tasks. These reports contain trend charts, bar charts, pie charts and other visual elements of key performance indicators, which help users intuitively understand the system operation status.

[0107] Users can customize report templates and select key indicators and time periods to be displayed to meet different monitoring needs.

[0108] 5. Key indicator trend display:

[0109] The system displays the changing trends of key performance indicators in the form of graphs, such as the change of average response time over time, the fluctuation of daily data update delay, etc. This helps users to identify potential problems and take timely measures.

[0110] Trend graphs can show long-term and short-term changes, helping users understand the stability and predictability of the system.

[0111] 6. Abnormal situation alarm:

[0112] The system automatically detects abnormal situations in data collection tasks through preset anomaly detection algorithms or user-defined rules and triggers an alarm mechanism. Alarm information can be sent to users via SMS, email, push notifications, etc. to ensure that users can respond in time.

[0113] Anomaly detection algorithms can identify abnormal patterns based on statistical methods (such as standard deviation, variance, etc.) or machine learning models. Users can also customize detection rules according to their own needs, such as when the value of a specific field exceeds the expected range.

[0114] In another preferred embodiment of the present invention, a central monitoring dashboard is provided for real-time monitoring of the status and performance of data acquisition tasks; when an abnormality or performance degradation is detected, the system automatically triggers an alarm and recommends optimization measures; and a fault diagnosis tool is further provided.

[0115] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A real-time process mining method based on data analysis, characterized in that: The following steps are involved: Provide a user interface that allows users to configure the input parameters of data collection tasks, including the target website URL and collection frequency; Implement predefined data collection, including web crawler module, for automatically accessing and crawling the content of target websites and preprocessing them; Provide API calling interface for obtaining data from third-party services or data providers; Based on database query instructions, it is used to directly extract data from the database and encrypt the data for transmission.

2. The real-time process mining method based on data analysis according to claim 1, characterized in that: The pre-processing process is as follows: Automatically identify and remove duplicate data, format errors, and unstructured text data; apply natural language processing technology to extract key information, including entity recognition and sentiment analysis.

3. The real-time process mining method based on data analysis according to claim 1, characterized in that: The encrypted transmission includes using SSL / TLS protocol encryption during data transmission and AES or RSA encryption during data storage; and implementing access control policies to authorize pre-selected users to access specific data.

4. The real-time process mining method based on data analysis according to claim 1, characterized in that: Automatically create data collection tasks based on the data model specified by the user, including determining the collection points and data fields; automatically configure the execution order and frequency of data collection tasks, and update data regularly or in real time; provide a visual interface to display the data model and data collection tasks for users to adjust and optimize the collection process.

5. The real-time process mining method based on data analysis according to claim 1, characterized in that: Provide development tools and APIs to support users in developing and debugging custom data acquisition modules; allow users to seamlessly integrate custom modules with existing data acquisition processes.

6. The real-time process mining method based on data analysis according to claim 1, characterized in that: Use window functions and grouping operations to analyze the browsing events and corresponding click events triggered by a specific user within a certain day; calculate the time difference between browsing events and click events, and determine the specific position of the user in the funnel conversion path; The calculation results are further aggregated to count the funnel conversion rate of a specific user group within a certain day, including the number of conversions between different states; finally, the results are grouped by event date, remaining date, and date difference, and the total number of times in each group is calculated.

7. The real-time process mining method based on data analysis according to claim 1, characterized in that: Analyze the execution status and performance indicators of historical data collection tasks, including response time and data accuracy; automatically adjust the parameters of data collection tasks based on the analysis results to increase or decrease the collection frequency.

8. The real-time process mining method based on data analysis according to claim 1, characterized in that: Record and analyze the execution time, response speed, and data accuracy of each data collection task, and define key performance indicators, including the average response time of data collection tasks, daily data update delays, and abnormal task occurrence rates; Utilize the report generation function provided by the low-code platform to regularly generate monitoring reports on the execution of historical data collection tasks, showing the trends and changes of key indicators; automatically detect and alarm abnormal situations in data collection tasks through preset anomaly detection algorithms or user-defined rules, including data loss, field errors, or frequent execution errors.

9. The real-time process mining method based on data analysis according to claim 8, characterized in that: Provides a central monitoring dashboard for real-time monitoring of the status and performance of data collection tasks; when anomalies or performance degradation are detected, the system automatically triggers alarms and recommends optimization measures; further provides fault diagnosis tools.

Images