A method for intelligent data analysis and automated operation

Abstract

The present invention discloses a method for intelligent data analysis and automated operation, which relates to the field of data processing technology. Configuration management, parsing business database, establishing data connection; in an offline environment, regularly executing bash scripts, business classification; integrating datav dashboards, reading configuration analysis dimensions, retrieving matching data, rendering as data dashboards of historical periodic statistics; setting the ranking proportion of job information; executing bi‑analyse.sh scripts, doing classification and sorting, synchronizing the sorting data of the offline environment to the specified environment, uploading to the online database, displaying it in the application, importing it into the dashboard platform, recording data updates and data extraction logs, and tracing data changes and extraction. The present invention realizes the internal data integration of small and medium-sized enterprises, reduces the difficulty of data extraction, has high security, improves real-time decision-making capabilities, and improves the overall work efficiency of the enterprise.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for intelligent data analysis and automated operation. Background Art

[0002] At present, there are many similar data intelligent analysis, data visualization and extraction platforms on the market, but they are not friendly enough for small and medium-sized enterprises. At the same time, the high maintenance costs and lack of support from professional developers make it daunting for small and medium-sized enterprises. The existing technology generally uses Kettle open source ETL tools, which are written in pure Java, green and do not require installation, and extract data efficiently and stably (data migration tools). It can run on Windows, Linux, and Unix, and extract data efficiently and stably; various data are put into a pot and then flow out in a specified format. However, there are the following disadvantages:

[0003] (1) Kettle stores all data sets in a centralized warehouse, which increases the complexity and difficulty of data maintenance. In addition, the data warehouse is single and cannot be extended or expanded.

[0004] (2) Lack of data visualization and data modification and deployment operations, making it impossible to provide customized services based on the unique needs of the enterprise;

[0005] (3) Detailed explanatory information cannot be provided during the data extraction phase, and an in-depth understanding of metadata is required to use this tool;

[0006] (4) Data extraction security cannot be verified compulsorily, and there is a risk of data theft.

[0007] In order to solve the above problems, it is particularly necessary to develop a method of intelligent data analysis and automated operation. Summary of the invention

[0008] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method for intelligent data analysis and automated operation, so as to realize internal data integration of small and medium-sized enterprises, reduce the difficulty of data extraction, have strong extension and scalability, high security, enhance real-time decision-making capabilities, improve the overall work efficiency of the enterprise, have low cost, and be easy to promote and use.

[0009] In order to achieve the above object, the present invention is implemented through the following technical solution: a method for intelligent data analysis and automated operation, the steps of which are:

[0010] (1) Use HOCON (Human-Optimized Config Object Notation) for configuration management;

[0011] (2) Use Java to read the file named database.conf (data source configuration), parse the business database in the current system, and establish a data operation connection;

[0012] (3) In an offline environment, a bash script is executed regularly to classify data fields related to similar businesses in different database tables into one table and store them in ES;

[0013] (4) Integrate the datav dashboard, read and configure the analysis dimension according to dimension.conf (analysis dimension configuration), retrieve the matching data through ES, and render it into a data dashboard with historical periodic statistics through datav;

[0014] (5) Set the ranking weight of job information by configuring sort-percent.conf;

[0015] (6) Dynamically adjust the proportion of each dimension based on the set data, execute the bi-analyse.sh script, and classify and sort the filtered result set based on the proportion algorithm;

[0016] (7) Use the data synchronization plug-in Maxwell to synchronize the sorting data of the offline environment to the specified environment, and finally upload it to the online database and display it in the application. Repeat the operations of steps (1) to (5) above to dynamically and flexibly adjust the job placement effect;

[0017] (8) The data generated by step (4) supports the export and extraction operation, which is used by the group to make decisions on the types and order of job placements, and is imported into the Kanban platform to execute step (6);

[0018] (9) Record data updates and data extraction logs to trace data changes and extractions.

[0019] Preferably, in step (2), the database.conf file is configured, wherein

[0020] #mysql

[0021] db-mysql-url: database link url

[0022] db-mysql-user: database login instructions

[0023] db-mysql-pwd: database login key

[0024] db-mysql-swich: whether to open mysql connection (0: no 1 yes)

[0025] #ES

[0026] db-es-user: connection instruction user

[0027] db-es-pwd: connection key

[0028] db-es-address: es deployment access address

[0029] db-es-swich: whether to open es connection

[0030] #mongodb

[0031] db-mg-user: connection instruction user

[0032] db-mg-pwd: connection key

[0033] db-mg-url: mongo database link address

[0034] db-mg-swich: whether to open mongodb connection

[0035] #HBase

[0036] db-hb-user: connection instruction user

[0037] db-hb-pwd: connection key

[0038] db-hb-url: HBase database link address

[0039] db-hb-swich: Whether to enable HBase connection.

[0040] Preferably, in step (4), dimension.conf is configured (analysis dimension configuration 0: No 1: Yes)

[0041] di-channel-swich: Whether the channel dimension is enabled

[0042] di-jobtype-swich: Whether the job type dimension is enabled

[0043] di-activeuser-swich: Whether the active user dimension is enabled

[0044] di-enrolleffect-swich: Whether the enrollment effect dimension is enabled

[0045] di-joblike-swich: Whether the job preference dimension is enabled

[0046] di-price-swich: Whether the job remuneration dimension is enabled

[0047] # ...other dimensions can be expanded.

[0048] Preferably, in step (5), sort-percent.conf is configured (position sorting involves metadata fields and weight configuration)

[0049] sort-publishtime-weight: the proportion of time publishing in the algorithm

[0050] sort-like-weight: the proportion of likes

[0051] sort-comment-weight: the proportion of comments

[0052] sort-channel-weight: channel distribution weight

[0053] sort-share-weight: share weight

[0054] # ...customize other weights.

[0055] Preferably, the business database in step (2) includes MySQL, TIDB, ES, and redis.

[0056] Preferably, the step (6) executes the bi-cancordance.sh script command, performs the cleaning of the above data regularly, integrates the business data into ES according to the configuration dimension, and quickly counts the metadata fields of various businesses. Access the service graphical interface, operate the business metadata required to be provided on the data dashboard page, and then the bi-analyse.sh script can be used in the background to intelligently count the metadata of the required items, and render them to the data dashboard through datav.

[0057] Preferably, the step (7) is updated to the application database through the Maxwell reverse data synchronization, and the data changes in real time to provide feedback and dynamically adjust the rules in real time.

[0058] The beneficial effects of the present invention are as follows: the present method realizes the internal data integration of small and medium-sized enterprises, reduces the difficulty of data extraction, ensures the real-time, accuracy and security of data acquisition, has strong extension and scalability, creates the enterprise's exclusive data intelligent analysis dashboard, improves real-time decision-making capabilities, improves the overall work efficiency of the enterprise, is simple and easy to use, reduces platform complexity, greatly improves usability, and is low-cost, making it easy to efficiently use data to provide a basis for enterprise strategy formulation, and has broad application prospects. DETAILED DESCRIPTION

[0059] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0060] This specific implementation adopts the following technical solution: a method for intelligent data analysis and automated operation, the steps of which are:

[0061] (1) HOCON (Human-Optimized Config Object Notation) is used for configuration management, which provides a more user-friendly reading format and is also a superset of JSON and .properties.

[0062] (2) Using Java to read database.conf (data source configuration), parse the business database in the current system, and establish a data operation connection; the business database includes mysql, TIDB, ES, redis, etc.;

[0063] (3) In an offline environment, a bash script is executed regularly to classify data fields related to similar businesses in different database tables into one table and store them in ES;

[0064] (4) Integrate the datav dashboard, read and configure the analysis dimension according to dimension.conf (analysis dimension configuration), retrieve the matching data through ES, and render it into a data dashboard with historical periodic statistics through datav;

[0065] (5) Set the ranking weight of job information by configuring sort-percent.conf;

[0066] (6) Dynamically adjust the proportion of each dimension based on the set data, execute the bi-analyse.sh script, and classify and sort the filtered result set based on the proportion algorithm;

[0067] (7) Use the data synchronization plug-in Maxwell to synchronize the sorting data of the offline environment to the specified environment, and finally upload it to the online database and display it in the application. Repeat the operations of steps (1) to (5) above to dynamically and flexibly adjust the job placement effect;

[0068] (8) The data generated by step (4) supports the export and extraction operation, which is used by the group to make decisions on the types and order of job placements, and is imported into the Kanban platform to execute step (6);

[0069] (9) The service interface operates data extraction, displays historical data, records data updates and data extraction logs, and traces data changes and extractions.

[0070] This specific implementation method uses java+bash script as the development language for building the underlying infrastructure of the platform and a series of background operations involving data change display and algorithm interaction; Angular is used as the front-end rendering interface for a series of click interaction operations on the foreground interactive interface.

[0071] This method configures the environment java, maven, ES and other related dependent environments. First, configure the database.conf file, where

[0072] #mysql

[0073] db-mysql-url: database link url

[0074] db-mysql-user: database login instructions

[0075] db-mysql-pwd: database login key

[0076] db-mysql-swich: whether to open mysql connection (0: no 1 yes)

[0077] #ES

[0078] db-es-user: connection instruction user

[0079] db-es-pwd: connection key

[0080] db-es-address: es deployment access address

[0081] db-es-swich: whether to open es connection

[0082] #mongodb

[0083] db-mg-user: connection instruction user

[0084] db—mg-pwd: connection key

[0085] db—mg-url: mongo database link address

[0086] db-mg-swich: whether to open mongodb connection

[0087] #HBase

[0088] db-hb-user: connection instruction user

[0089] db-hb-pwd: connection key

[0090] db-hb-url: HBase database link address

[0091] db-hb-swich: Whether to enable HBase connection.

[0092] Then configure dimension.conf (analysis dimension configuration 0: No 1: Yes)

[0093] di-channel-swich: Whether the channel dimension is enabled

[0094] di-jobtype-swich: Whether the job type dimension is enabled

[0095] di-activeuser-swich: Whether the active user dimension is enabled

[0096] di-enrolleffect-swich: Whether the enrollment effect dimension is enabled

[0097] di-joblike-swich: Whether the job preference dimension is enabled

[0098] di-price-swich: Whether the job remuneration dimension is enabled

[0099] # ...other dimensions can be expanded.

[0100] You also need to configure sort-percent.conf (position sorting involves metadata fields and weight configuration)

[0101] sort-publishtime-weight: the proportion of time publishing in the algorithm

[0102] sort-like-weight: the proportion of likes

[0103] sort-comment-weight: the proportion of comments

[0104] sort-channel-weight: channel distribution weight

[0105] sort-share-weight: share weight

[0106] # ...customize other weights.

[0107] It is worth noting that you can log in to the server through the ssh tool, package the code to generate a jar file and upload it to the server directory: / data / BI / code / , execute the command to start the service, and use zabbix to monitor the service port status; synchronize the maxwell service, monitor the required synchronization data, start the service, and synchronize it in real time to the database pointed to by the deployment service.

[0108] Execute the bi-cancordance.sh script command to perform the above data cleaning regularly, integrate the business data into ES according to the configuration dimension, and quickly count the metadata fields of various businesses. Access the service graphical interface, operate the business metadata required on the data dashboard page, and then use the bi-analyse.sh script in the background to intelligently count the metadata required for each item, and render it to the data dashboard through datav.

[0109] In addition, the dashboard interface can adjust the job data display sorting rules for business dimension data based on data effect analysis (visual configuration weight or adding existing metadata as statistical weight), and update the application database through Maxwell reverse data synchronization. The data changes in real time to feedback the effect and adjust the rules dynamically in real time.

[0110] The system functions of this specific implementation method are: ① combing internal data, such as effective data integration and splitting of data clusters composed of relational business database MySQL and non-relational databases (setting analysis categories from custom dimensions such as job placement channels, registration effects, active users, job types, and preferences);

[0111] ②Analyze the source database, split the database table fields, and group the database table fields involved in the same business;

[0112] ③ Regularly clean the data, desensitize sensitive information and other important information in a timely manner, and synchronize the data to an offline environment for internal analysis;

[0113] ④Configure the business category of data analysis;

[0114] ⑤Configure the main dimensions of the database tables involved in data analysis and the fields after aggregation, and use datav to display data analysis reports;

[0115] ⑥Support customized job data sorting and uploading, and timely adjust sorting algorithms and information based on dashboard analysis data;

[0116] ⑦Support encrypted export of analysis data;

[0117] ⑧ Historical data analysis, data extraction and other traceability monitoring.

[0118] The technical advantages of this specific implementation method are: (1) the internal data of the enterprise is interconnected and aggregated into its own data platform to integrate data, set dimensional intelligent analysis, and summarize various analysis and statistical dashboards;

[0119] (2) It supports complex sorting algorithms for analyzed data and can adjust data based on algorithm rules according to real-time data analysis dashboards, with immediate results;

[0120] (3) Provide functions such as data dashboard, data visualization extraction, hot data extraction frequency, data extraction trend, and extraction log information recording;

[0121] (4) Support data analysis support system based on multiple database sources;

[0122] (5) This system reduces manual learning, maintenance and operating costs and improves work efficiency.

[0123] This specific implementation method uses a platform-based model to support enterprises in efficiently obtaining various business data, thereby supporting enterprises in making various decisions; the dimension weighting and sorting algorithm of the data can be updated through the background of the visual dashboard, and the updated data changes can be seen immediately, which greatly improves the real-time decision-making ability; the platform construction cost of this method is low, the portability is high, and the maintenance cost is low. It not only supports various business data but also supports the extraction of embedded behavior data; it also supports that the extracted data can be fully customized, and the platform supports the display of the platform's existing data dimensions in the form of wiki knowledge archiving.

[0124] In summary, this method realizes the internal data integration of small and medium-sized enterprises, reduces the difficulty of data extraction, and ensures the real-time, accuracy and security of data acquisition, creates the company's exclusive data intelligent analysis dashboard, and can instantly configure the rectification data display rules and dynamically adjust them according to the results; so that relevant personnel do not need to spend too much energy on the data extraction process, only need to select and filter the extraction dimensions through the visual interface operation to obtain the required extracted data in real time; at the same time, during the platform use period, only the wiki document corresponding to the database table metadata field needs to be designed in advance, which is simple and easy to use, the complexity is greatly reduced, the usability is greatly improved, the manual learning cost is reduced, the overall work efficiency of the enterprise is improved, and the data is efficiently used to provide a basis for the formulation of corporate strategy, which has broad market application prospects.

[0125] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for intelligent data analysis and automated operation, characterized in that: The steps are: (1) Use HOCON for configuration management; (2) Use Java to read the database.conf file, parse the business database in the current system, and establish a data operation connection; (3) In an offline environment, a bash script is executed regularly to classify data fields related to similar businesses in different database tables into one table and store them in ES; (4) Integrate datav dashboard, read and configure analysis dimensions according to dimension.conf, retrieve matching data through ES, and render it into a data dashboard with historical periodic statistics through datav; (5) Set the ranking weight of job information by configuring sort-percent.conf; (6) Dynamically adjust the proportion of each dimension based on the set data, execute the bi-analyse.sh script, and classify and sort the filtered result set based on the proportion algorithm; (7) Use the data synchronization plug-in Maxwell to synchronize the sorting data of the offline environment to the specified environment, and finally upload it to the online database and display it in the application. Repeat the operations of steps (1) to (5) above to dynamically and flexibly adjust the job placement effect; (8) The data generated by step (4) supports the export and extraction operation, which is used by the group to make decisions on the types and order of job placements, and is imported into the Kanban platform to execute step (6); (9) Record data updates and data extraction logs to trace data changes and extractions.

2. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: In step (2) above, configure the database.conf file, where #mysql db-mysql-url: database link url db-mysql-user: database login instructions db-mysql-pwd: database login key db-mysql-swich: whether to open mysql connection #ES db-es-user: connection instruction user db-es-pwd: connection key db-es-address: es deployment access address db-es-swich: whether to open es connection #mongodb db-mg-user: connection instruction user db-mg-pwd: connection key db-mg-url: mongo database link address db-mg-swich: whether to open mongodb connection #HBase db-hb-user: connection instruction user db-hb-pwd: connection key db-hb-url: HBase database link address db-hb-swich: Whether to enable HBase connection.

3. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: Configure dimension.conf in step (4) di-channel-swich: Whether the channel dimension is enabled di-jobtype-swich: Whether the job type dimension is enabled di-activeuser-swich: Whether the active user dimension is enabled di-enrolleffect-swich: Whether the enrollment effect dimension is enabled di-joblike-swich: Whether the job preference dimension is enabled di-price-swich: Whether the job remuneration dimension is enabled #......Extension to other dimensions.

4. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: Configure sort-percent.conf in step (5) sort-publishtime-weight: the proportion of time publishing in the algorithm sort-like-weight: the proportion of likes sort-comment-weight: the proportion of comments sort-channel-weight: channel distribution weight sort-share-weight: share weight #......Customize other weights.

5. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: The business database in step (2) includes mysql, TIDB, ES, and redis.

6. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: The step (6) executes the bi-cancordance.sh script command, regularly performs the cleaning of the above data, and integrates the business data into ES according to the configuration dimension, and quickly counts the metadata fields of various businesses.

7. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: The step (6) intelligently counts the metadata of each item required in the background through the bi-analyse.sh script, and renders it to the data dashboard through datav.

8. The method of data intelligent analysis and automated operation according to claim 1, characterized in that: The step (7) is updated to the application database through the Maxwell reverse data synchronization, and the data changes in real time to provide feedback and dynamically adjust the rules in real time.