Data report generation method and device, computer device, and storage medium

Abstract

This application belongs to the field of big data analysis and is applied to the field of data report generation. It relates to a data report generation method, apparatus, computer equipment, and storage medium. The method includes: collecting initial data and storing it in a relational database; sending the initial data to a HiVe data warehouse based on a Druid data connection pool; performing offline ETL processing on the initial data and displaying it on an analysis interface; obtaining user analysis requirements and generating HQL execution statements; executing the analysis plan and obtaining data analysis results; and displaying the data analysis results on the data analysis interface. The method uses a Druid connection pool to connect the relational database and the HiVe data warehouse. The initial data is stored in the relational database, and during analysis, the data in the HiVe data warehouse is processed to achieve data reuse. Simultaneously, the Druid connection pool improves data transmission speed and ensures timeliness.

Description

Technical Field

[0001] This application relates to the fields of big data and data analysis technology, and in particular to a data report generation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of the Internet, business data has grown explosively. How to maximize the value of data is a question we need to consider. Statistical analysis of data to generate data reports can help uncover the value hidden in the data.

[0003] Currently, the industry generally faces the following problems with data analysis: siloed development is used for data report development, where reports are generated one by one for each data analysis request, making data reuse impossible, wasting a lot of manpower and resources, and resulting in very poor timeliness and flexibility. Therefore, while some open-source components, such as PRESTO, have solved the problem of wasted manpower and resources, when the data volume reaches a certain level, the rendering of reports generated from the data becomes very slow, timeliness cannot be guaranteed, and data reuse is not possible. Summary of the Invention

[0004] The purpose of this application is to provide a data report generation method, apparatus, computer equipment, and storage medium to facilitate data reuse and improve the timeliness of data analysis.

[0005] To address the aforementioned technical problems, this application provides a data report generation method, employing the following technical solution:

[0006] A method for generating data reports includes the following steps:

[0007] Collect real-time and offline data to be analyzed as initial data and store them in a preset relational database;

[0008] The initial data is retrieved from the relational database using the Druid data connection pool and sent to the Hive data warehouse.

[0009] The initial data is processed offline using the Hive data warehouse and then displayed on a preset data analysis interface.

[0010] The system acquires the data analysis dimensions, data display metrics, data filtering conditions, and metric correlation algorithms selected by the user through click events on the data analysis interface.

[0011] Based on the data analysis dimensions, data display indicators, data filtering conditions, and indicator association algorithms, HQL execution statements are generated.

[0012] The HQL execution statement is converted into a data analysis plan and stored in the HDFS file system based on the Hadoop cluster. The data analysis plan is then executed to obtain a data analysis report.

[0013] The data analysis report is displayed in a preset area within the data analysis interface.

[0014] Furthermore, the step of retrieving the initial data from the relational database based on the Druid data connection pool and sending it to the Hive data warehouse specifically includes:

[0015] The initial data is retrieved from the relational database using the Druid data connection pool, and the metadata of the initial data is obtained from a preset configuration file. The initial data and the metadata are then sent together to the Hive data warehouse.

[0016] The metadata is used to indicate the storage location of the initial data in the relational database, including the table name of the table corresponding to the initial data, the column names in the table, and the partition information in the table.

[0017] Furthermore, the step of performing offline ETL processing on the initial data based on the Hive data warehouse specifically includes:

[0018] The initial data is loaded into a preset directory within the HDFS file system based on the Hadoop cluster;

[0019] After loading is complete, the directory information corresponding to the metadata of the initial data is obtained, and the directory information is sent to the relational database for storage.

[0020] Furthermore, the step of loading the data source into a preset directory within the HDFS file system based on the Hadoop cluster specifically includes:

[0021] Several sibling directories are pre-set in the HDFS file system. During the loading process, each unit of data in the initial data is loaded into the several sibling directories according to its corresponding table name or partition information.

[0022] Several subdirectories of the same level are set under the aforementioned several same-level directories. When loading, the data of each cell with different column names under the same table name or different column names under the same partition information are loaded separately into the aforementioned several same-level subdirectories.

[0023] The step of obtaining the directory information corresponding to the metadata of the initial data after loading is completed specifically includes:

[0024] After the loading is completed, the directory information of each unit data in the initial data is obtained and loaded differently according to its corresponding table name or partition information, and is used as the first directory information;

[0025] After the loading is completed, the directory information for loading different column names under the same table name or different column names under the same partition information is obtained and loaded separately, and is used as the second directory information;

[0026] The first directory information and the second directory information are used as the directory information corresponding to the metadata of the initial data.

[0027] Furthermore, the step of generating HQL execution statements based on the data analysis dimensions, data display indicators, data filtering conditions, and indicator association algorithms specifically includes:

[0028] Obtain the directory information corresponding to the metadata of the initial data as the execution library;

[0029] The data analysis dimensions, data display metrics, data filtering conditions, and metric association algorithms are used as execution conditions, execution results, constraints on execution results, and association relationships of execution results, respectively, to jointly complete the construction of HQL execution statements with the execution library.

[0030] Furthermore, the step of obtaining the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithm selected by the user through click events on the data analysis interface specifically includes:

[0031] Before the step of obtaining the data analysis dimension, data display metrics, data filtering conditions, and metric correlation algorithm selected by the user through a click event on the data analysis interface, the following steps are included:

[0032] The data analysis interface includes a first click field, a second click field, a filter condition setting field, and a correlation algorithm setting field. The first click field includes a first search box and a list of optional analysis dimensions. The second click field includes a second search box and a list of optional display indicators. The filter condition setting field includes a filter condition customization box and a list of optional filter conditions. The correlation algorithm setting field includes a correlation algorithm customization box and a list of optional correlation algorithms.

[0033] The steps of acquiring the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithms selected by the user through click events on the data analysis interface specifically include:

[0034] Based on the preset click monitoring component, the system identifies and obtains the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user in the first click domain, the second click domain, the filter condition setting domain, and the association algorithm setting domain.

[0035] Furthermore, the step of identifying and acquiring the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user in the first click domain, second click domain, filter condition setting domain, and association algorithm setting domain based on the preset click monitoring component specifically includes:

[0036] Activate the preset click monitoring component;

[0037] If a click request is detected in the first click domain, the click request is parsed, and at least one data analysis dimension from the list of selectable analysis dimensions pointed to by the click request is obtained based on the parsing result;

[0038] If a click request is detected in the second click domain, the click request is parsed, and at least one data display indicator from the list of optional display indicators pointed to by the click request is obtained based on the parsing result;

[0039] If a click request is detected in the filter condition setting field, the click request is parsed, and at least one data filter condition pointed to by the click request is obtained based on the parsing result. The data filter condition can be customized by the user in the filter condition customization box, or can be selected by the user in the filter condition selectable list.

[0040] If a click request is detected in the association algorithm setting field, the click request is parsed, and at least one indicator association algorithm pointed to by the click request is obtained based on the parsing result. The indicator association algorithm can be customized by the user in the association algorithm customization box, or can be selected by the user in the association algorithm selectable list.

[0041] To address the aforementioned technical problems, this application also provides a data report generation device, which employs the following technical solution:

[0042] A data report generation device, comprising:

[0043] The data acquisition module is used to collect real-time and offline data to be analyzed as initial data and store it in a preset relational database.

[0044] The data forwarding module is used to obtain the initial data from the relational database based on the Druid data connection pool and send it to the Hive data warehouse;

[0045] The interface initialization module is used to perform offline ETL processing on the initial data based on the Hive data warehouse and display it on the preset data analysis interface.

[0046] The analysis instruction acquisition module is used to acquire the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithms selected by the user through click events on the data analysis interface.

[0047] The execution statement generation module is used to generate HQL execution statements based on the data analysis dimensions, data display indicators, data filtering conditions, and indicator association algorithms.

[0048] The analysis result acquisition module is used to convert the HQL execution statement into a data analysis plan and store it in the HDFS file system based on the Hadoop cluster, execute the data analysis plan, and obtain the data analysis results.

[0049] The analysis results display module is used to display the data analysis report in a preset area within the data analysis interface.

[0050] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0051] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data report generation method described above.

[0052] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0053] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the data report generation method described above.

[0054] Compared with the prior art, the embodiments of this application have the following main advantages:

[0055] The data report generation method described in this application involves: collecting initial data and storing it in a relational database; sending the initial data to a Hive data warehouse using a Druid data connection pool; performing offline ETL processing on the initial data and displaying it on an analysis interface; obtaining user analysis requirements and generating HQL execution statements; executing the analysis plan and obtaining data analysis results; and displaying the data analysis results on the data analysis interface. The method uses a Druid connection pool to connect the relational database and the Hive data warehouse. The initial data is stored in the relational database, while the data in the Hive data warehouse is processed during analysis to achieve data reuse. Simultaneously, the Druid connection pool improves data transmission speed and ensures timeliness. Attached Figure Description

[0056] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0058] Figure 2 A flowchart of an embodiment of the data report generation method according to this application;

[0059] Figure 3 yes Figure 2 A flowchart of a specific implementation of step 203 shown;

[0060] Figure 4 yes Figure 3 A flowchart of a specific implementation of step 301 shown;

[0061] Figure 5 yes Figure 2 A flowchart of a specific implementation of step 207 shown;

[0062] Figure 6 This is a schematic diagram of the structure of an embodiment of the data report generation apparatus according to this application;

[0063] Figure 7 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0068] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0069] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0070] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0071] It should be noted that the data report generation method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the data report generation device is generally set in the server / terminal device.

[0072] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0073] Continue to refer to Figure 2 A flowchart of an embodiment of the data report generation method according to this application is shown. The data report generation method includes the following steps:

[0074] Step 201: Collect real-time and offline data to be analyzed as initial data and store them in a preset relational database.

[0075] In this embodiment, the initial data is stored in a relational database, ensuring that the initial data in the database is not modified. Instead, it is sent to the Hive data warehouse as initialization data, enabling the reuse of the initial data.

[0076] Step 202: Obtain the initial data from the relational database based on the Druid data connection pool and send it to the Hive data warehouse.

[0077] In this embodiment, the step of obtaining the initial data from the relational database based on the Druid data connection pool and sending it to the Hive data warehouse specifically includes: obtaining the initial data from the relational database based on the Druid data connection pool, obtaining the metadata of the initial data from a preset configuration file, and sending the initial data and the metadata together to the Hive data warehouse. The metadata is used to indicate the storage location of the initial data in the relational database, including the table name of the table corresponding to the initial data, the column names in the table, and the partition information in the table.

[0078] In this embodiment, the initial data is sent to the Hive data warehouse using the Druid database connection pool. The Druid database connection pool adopts a distributed connection method, and the Hive data warehouse adopts a distributed caching method, which ensures the timeliness of data transmission. Therefore, it also improves the timeliness of data analysis to a certain extent.

[0079] Step 203: Perform offline ETL processing on the initial data based on the Hive data warehouse and display it on the preset data analysis interface.

[0080] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific implementation of step 203 shown includes the following steps:

[0081] Step 301: Load the initial data into a preset directory within the HDFS file system based on the Hadoop cluster;

[0082] By using a Hadoop cluster, and combining clustering with distributed caching, the speed of data transmission is further improved, while the coupling of system processing is reduced.

[0083] Step 302: After loading is complete, obtain the directory information corresponding to the metadata of the initial data, and send the directory information to the relational database for storage.

[0084] By sending the directory information to the relational database for storage, the Hive data warehouse only stores the data and not the metadata such as table names, column names, and partition information. This reduces the time spent executing query statements and improves the timeliness of data analysis to some extent.

[0085] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific implementation of step 301 shown includes the following steps:

[0086] Step 401: Several sibling directories are pre-set in the HDFS file system. During the loading process, each unit of data in the initial data is loaded into the several sibling directories according to its corresponding table name or partition information.

[0087] By caching data from different table names or different partitions in several directories at the same level, the data caching becomes more standardized, making it easier to render data during data analysis.

[0088] Step 402: Set up several subdirectories at the same level under the several same-level directories. When loading, load the data of each unit with different column names under the same table name or different column names under the same partition information into the several subdirectories at the same level.

[0089] By caching data with the same table name or different column names under the same partition into several sibling subdirectories, the data caching becomes more standardized, making it easier to render data during data analysis.

[0090] The step of obtaining the directory information corresponding to the metadata of the initial data after loading is completed includes:

[0091] Step 403: After the loading is completed, obtain the directory information of each unit data in the initial data according to its corresponding table name or partition information, and use it as the first directory information.

[0092] Step 404: After the loading is completed, obtain the directory information for different loading of each unit data under the same table name or under the same partition information, and use it as the second directory information.

[0093] Step 405: Use the first directory information and the second directory information as the directory information corresponding to the metadata of the initial data.

[0094] By sending directory information to the relational database for storage, the Hive data warehouse only stores data, reducing the time spent executing query statements and improving the timeliness of data analysis to some extent.

[0095] Step 204: Obtain the data analysis dimension, data display indicators, data filtering conditions, and indicator association algorithm selected by the user through click events on the data analysis interface.

[0096] In this embodiment, the step of obtaining the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithm selected by the user through a click event on the data analysis interface specifically includes:

[0097] Before the step of obtaining the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user through click events on the data analysis interface, a first click domain, a second click domain, a filtering condition setting domain, and an association algorithm setting domain are preset in the data analysis interface. The first click domain includes a first search box and a list of selectable analysis dimensions; the second click domain includes a second search box and a list of selectable display indicators; the filtering condition setting domain includes a custom filtering condition box and a list of selectable filtering conditions; and the association algorithm setting domain includes a custom association algorithm box and a list of selectable association algorithms. The step of obtaining the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user through click events on the data analysis interface specifically includes: based on a preset click monitoring component, identifying and obtaining the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user in the corresponding fields of the first click domain, second click domain, filtering condition setting domain, and association algorithm setting domain.

[0098] In this embodiment, the step of identifying and obtaining the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user in the first click domain, second click domain, filter condition setting domain, and association algorithm setting domain based on the preset click monitoring component specifically includes: activating the preset click monitoring component; if a click request is detected in the first click domain, parsing the click request and obtaining at least one data analysis dimension from the list of selectable analysis dimensions pointed to by the click request based on the parsing result; if a click request is detected in the second click domain, parsing the click request and obtaining at least one data analysis dimension from the list of selectable display indicators pointed to by the click request based on the parsing result. At least one data display metric; if a click request is detected in the filter condition setting field, the click request is parsed, and at least one data filter condition pointed to by the click request is obtained based on the parsing result, wherein the data filter condition can be customized by the user in the filter condition customization box, or can be selected by the user in the filter condition selectable list; if a click request is detected in the association algorithm setting field, the click request is parsed, and at least one indicator association algorithm pointed to by the click request is obtained based on the parsing result, wherein the indicator association algorithm can be customized by the user in the association algorithm customization box, or can be selected by the user in the association algorithm selectable list.

[0099] By clicking on the monitoring component, also known as the listening component, users can determine the dimensions, indicators, filtering conditions, and correlation algorithms for data analysis. Users can obtain data dimensions, indicators, filtering conditions, and correlation algorithms in real time by dragging and dropping or customizing according to their needs, thus satisfying their flexibility and autonomy in data analysis.

[0100] Step 205: Generate HQL execution statements based on the data analysis dimensions, data display indicators, data filtering conditions, and indicator association algorithms.

[0101] In this embodiment, the step of generating HQL execution statements based on the data analysis dimensions, data display metrics, data filtering conditions, and metric association algorithms specifically includes: obtaining the directory information corresponding to the metadata of the initial data as an execution library; using the data analysis dimensions, data display metrics, data filtering conditions, and metric association algorithms as execution conditions, execution results, constraints on execution results, and association relationships of execution results, respectively, together with the execution library, to complete the construction of HQL execution statements.

[0102] Assuming the directory information is " / wh / application", named database A, the data analysis dimensions include the number of orders signed today (B) and the number of orders signed yesterday (C), the data display indicators include the number of household orders signed (D), the data filtering conditions include time filtering restrictions on single day and previous day's time period (E), and the indicator association algorithm is the increment function F(D) of the number of orders signed today compared to the previous day. Then the corresponding HQL execution statement is "select F(D), D from A where B,C group by D having F(D) order by D.idasc", which means querying the increment function F(D) of the number of household orders signed today (B) and the number of orders signed yesterday (C) in database A, and sorting them by 'asc' according to the corresponding id of the household orders signed.

[0103] Data can be queried directly in the Hive data warehouse by executing HQL statements, without having to convert HQL statements into SQL statements to query data from a relational database. This ensures that data is not queried from the relational database and guarantees the reuse of the initial data.

[0104] Step 206: Convert the HQL execution statement into a data analysis plan and store it in the HDFS file system based on the Hadoop cluster. Execute the data analysis plan to obtain a data analysis report.

[0105] Step 207: Display the data analysis report in a preset area within the data analysis interface.

[0106] In this embodiment, the step of displaying the data analysis report in a preset area within the data analysis interface specifically includes: determining whether a preset specific display plugin exists; if not, displaying the data analysis report in a list format within the preset area of ​​the data analysis interface; if so, displaying the data analysis report in any one or more of the pie chart, histogram, and line chart formats within the preset area of ​​the data analysis interface based on the type of the preset specific display plugin, wherein the preset specific display plugin includes, but is not limited to, pie chart, histogram, and line chart display plugins.

[0107] Continue to refer to Figure 5 , Figure 5 yes Figure 2 A flowchart of a specific implementation of step 207 shown includes the following steps:

[0108] Step 501: Determine if a specific preset display plugin exists;

[0109] Step 502: If it does not exist, the data analysis report is displayed in a list format in a preset area of ​​the data analysis interface.

[0110] Step 503: If present, the data analysis report is displayed in a preset area of ​​the data analysis interface in any one or more ways, such as a pie chart, histogram, or line chart, based on the type of the preset specific display plugin.

[0111] By determining whether preset graphic plugins are used when displaying data analysis results, flexible display of data analysis results can be achieved, meeting users' personalized and diverse needs.

[0112] This application involves collecting initial data and storing it in a relational database; sending the initial data to a Hive data warehouse using a Druid data connection pool; performing offline ETL processing on the initial data and displaying it on the analysis interface; obtaining user analysis requirements and generating HQL execution statements; executing the analysis plan and obtaining data analysis results; and displaying the data analysis results on the data analysis interface. The application uses a Druid connection pool to connect the relational database and the Hive data warehouse, storing the initial data in the relational database while processing the data in the Hive data warehouse during analysis to achieve data reuse. Simultaneously, the Druid connection pool improves data transmission speed and ensures timeliness.

[0113] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0114] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0115] For example, in this embodiment of the application, a neural network model in artificial intelligence can be used to determine the multidimensional dimensions of the data to be analyzed. The multidimensional dimensions can be set as the data analysis dimensions of the data to be analyzed, and their corresponding feature values ​​can be set as data analysis indicators. At the same time, an indicator association algorithm can be generated based on artificial intelligence.

[0116] Further reference Figure 6 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data report generation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0117] like Figure 6 As shown, the data report generation device 600 described in this embodiment includes: a data acquisition module 601, a data forwarding module 602, an interface initialization module 603, an analysis instruction acquisition module 604, an execution statement generation module 605, an analysis result acquisition module 606, and an analysis result display module 607. Wherein:

[0118] The data acquisition module 601 is used to collect real-time data and offline data to be analyzed as initial data and store them in a preset relational database.

[0119] The data forwarding module 602 is used to obtain the initial data from the relational database based on the Druid data connection pool and send it to the Hive data warehouse;

[0120] The interface initialization module 603 is used to perform offline ETL processing on the initial data based on the Hive data warehouse and display it on the preset data analysis interface.

[0121] The analysis instruction acquisition module 604 is used to acquire the data analysis dimension, data display indicators, data filtering conditions and indicator association algorithm selected by the user through click events on the data analysis interface.

[0122] The execution statement generation module 605 is used to generate HQL execution statements based on the data analysis dimensions, data display indicators, data filtering conditions and indicator association algorithms.

[0123] The analysis result acquisition module 606 is used to convert the HQL execution statement into a data analysis plan and store it in the HDFS file system based on the Hadoop cluster, execute the data analysis plan, and obtain the data analysis results.

[0124] The analysis results display module 607 is used to display the data analysis report in a preset area within the data analysis interface.

[0125] This application collects initial data through a data acquisition module and stores it in a relational database; it then sends the initial data to a Hive data warehouse via a data forwarding module based on a Druid data connection pool; the initial data undergoes offline ETL processing through an interface initialization module and is displayed on the analysis interface; user analysis requirements are obtained through an analysis instruction acquisition module and an execution statement generation module, generating HQL execution statements; the analysis plan is executed through an analysis result acquisition module to obtain data analysis results; and the data analysis results are displayed on the data analysis interface through an analysis result display module. A Druid connection pool is used to connect the relational database and the Hive data warehouse. The initial data is stored in the relational database, while the data in the Hive data warehouse is processed during analysis to achieve data reuse. Simultaneously, the Druid connection pool is used to improve data transmission speed and ensure timeliness.

[0126] In this embodiment, the interface initialization module includes a loading submodule and a caching submodule. The loading submodule loads the initial data into a preset directory within the HDFS file system based on the Hadoop cluster. The caching submodule, after loading is complete, retrieves the directory information corresponding to the metadata of the initial data and sends the directory information to the relational database for storage.

[0127] In some optional implementations of this embodiment, the loading submodule includes a first setting unit and a second setting unit. The first setting unit is used to pre-set several sibling directories in the HDFS file system. During loading, it loads the data of each unit in the initial data into the several sibling directories according to their corresponding table names or partition information. The second setting unit is used to set several sibling subdirectories under the several sibling directories. During loading, it loads the data of each unit with different column names under the same table name or different column names under the same partition information into the several sibling subdirectories.

[0128] In some optional implementations of this embodiment, the caching submodule includes a first directory information acquisition unit and a second directory information acquisition unit. The first directory information acquisition unit, after the loading is complete, acquires directory information for differentiating and loading each unit data in the initial data according to its corresponding table name or partition information, as the first directory information. The second directory information acquisition unit, after the loading is complete, acquires directory information for differentiating and loading each unit data with different column names under the same table name or different column names under the same partition information, as the second directory information.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0130] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0131] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 7 , Figure 7This is a basic structural block diagram of the computer device in this embodiment.

[0132] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected via a system bus. It should be noted that only the computer device 7 with components 71-73 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0133] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0134] The memory 71 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 7. Of course, the memory 71 may include both the internal storage unit and its external storage device of the computer device 7. In this embodiment, the memory 71 is typically used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions for data report generation methods. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or will be output.

[0135] In some embodiments, the processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 72 is typically used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to execute computer-readable instructions stored in the memory 71 or to process data, for example, to execute computer-readable instructions for the data report generation method.

[0136] The network interface 73 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 7 and other electronic devices.

[0137] The computer device proposed in this embodiment belongs to the field of big data analysis technology. This application collects initial data and stores it in a relational database; sends the initial data to a Hive data warehouse based on a Druid data connection pool; performs offline ETL processing on the initial data and displays it on the analysis interface; obtains user analysis requirements and generates HQL execution statements; executes the analysis plan and obtains data analysis results; displays the data analysis results on the data analysis interface. A Druid connection pool is used to connect the relational database and the Hive data warehouse. The initial data is stored in the relational database, and during analysis, the data in the Hive data warehouse is processed to achieve data reuse. Simultaneously, the Druid connection pool is used to improve data transmission speed and ensure timeliness.

[0138] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the data report generation method described above.

[0139] The computer-readable storage medium proposed in this embodiment belongs to the field of big data analytics technology. This application involves collecting initial data and storing it in a relational database; sending the initial data to a Hive data warehouse using a Druid data connection pool; performing offline ETL processing on the initial data and displaying it on the analysis interface; obtaining user analysis requirements and generating HQL execution statements; executing the analysis plan and obtaining data analysis results; and displaying the data analysis results on the data analysis interface. The Druid connection pool connects the relational database and the Hive data warehouse. The initial data is stored in the relational database, while the data in the Hive data warehouse is processed during analysis, achieving data reuse. Simultaneously, the Druid connection pool improves data transmission speed and ensures timeliness.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0141] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for generating data reports, characterized in that, Includes the following steps: Collect real-time and offline data to be analyzed as initial data and store them in a preset relational database; The initial data is obtained from the relational database based on the Druid data connection pool and sent to the Hive data warehouse, where the Hive data warehouse adopts a distributed caching method; The initial data is processed offline using the Hive data warehouse and then displayed on a preset data analysis interface. Specifically, the steps of processing the initial data offline using the Hive data warehouse include: loading the initial data into a preset directory within the HDFS file system of the Hadoop cluster; after loading, obtaining the directory information corresponding to the metadata of the initial data and sending the directory information to the relational database for storage. The system acquires the data analysis dimensions, data display metrics, data filtering conditions, and metric correlation algorithms selected by the user through click events on the data analysis interface. Based on the data analysis dimensions, data display metrics, data filtering conditions, and metric association algorithms, an HQL execution statement is generated. Specifically, the step of generating the HQL execution statement based on the data analysis dimensions, data display metrics, data filtering conditions, and metric association algorithms includes: The directory information corresponding to the metadata of the initial data is obtained as the execution library; the data analysis dimensions, data display indicators, data filtering conditions, and indicator association algorithms are respectively used as execution conditions, execution results, constraints on execution results, and association relationships of execution results, and together with the execution library, the HQL execution statement is constructed. The HQL execution statement is converted into a data analysis plan and stored in the HDFS file system based on the Hadoop cluster. The data analysis plan is then executed to obtain a data analysis report. The data analysis report is displayed in a preset area within the data analysis interface.

2. The data report generation method according to claim 1, characterized in that, The step of retrieving the initial data from the relational database based on the Druid data connection pool and sending it to the Hive data warehouse specifically includes: The initial data is retrieved from the relational database using the Druid data connection pool, and the metadata of the initial data is obtained from a preset configuration file. The initial data and the metadata are then sent together to the Hive data warehouse. The metadata is used to indicate the storage location of the initial data in the relational database, including the table name of the table corresponding to the initial data, the column names in the table, and the partition information in the table.

3. The data report generation method according to claim 1, characterized in that, The step of loading the initial data into a preset directory within the HDFS file system based on the Hadoop cluster specifically includes: Several sibling directories are pre-set in the HDFS file system. During the loading process, each unit of data in the initial data is loaded into the several sibling directories according to its corresponding table name or partition information. Several subdirectories of the same level are set under the aforementioned several same-level directories. When loading, the data of each cell with different column names under the same table name or different column names under the same partition information are loaded separately into the aforementioned several same-level subdirectories. The step of obtaining the directory information corresponding to the metadata of the initial data after loading is completed specifically includes: After the loading is completed, the directory information of each unit data in the initial data is obtained and loaded differently according to its corresponding table name or partition information, and is used as the first directory information; After the loading is completed, the directory information for loading different column names under the same table name or different column names under the same partition information is obtained and loaded separately, and is used as the second directory information; The first directory information and the second directory information are used as the directory information corresponding to the metadata of the initial data.

4. The data report generation method according to claim 1, characterized in that, The steps of obtaining the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithms selected by the user through click events on the data analysis interface specifically include: Before the step of obtaining the data analysis dimension, data display metrics, data filtering conditions, and metric correlation algorithm selected by the user through a click event on the data analysis interface, the following steps are included: The data analysis interface includes a first click field, a second click field, a filter condition setting field, and a correlation algorithm setting field. The first click field includes a first search box and a list of optional analysis dimensions. The second click field includes a second search box and a list of optional display indicators. The filter condition setting field includes a filter condition customization box and a list of optional filter conditions. The correlation algorithm setting field includes a correlation algorithm customization box and a list of optional correlation algorithms. The steps of acquiring the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithms selected by the user through click events on the data analysis interface specifically include: Based on the preset click monitoring component, the system identifies and obtains the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user in the first click domain, the second click domain, the filter condition setting domain, and the association algorithm setting domain.

5. The data report generation method according to claim 4, characterized in that, The steps of identifying and acquiring the data analysis dimension, data display indicator, data filtering condition, and indicator association algorithm selected by the user in the first click domain, second click domain, filter condition setting domain, and association algorithm setting domain based on the preset click monitoring component specifically include: Activate the preset click monitoring component; If a click request is detected in the first click domain, the click request is parsed, and at least one data analysis dimension from the list of selectable analysis dimensions pointed to by the click request is obtained based on the parsing result; If a click request is detected in the second click domain, the click request is parsed, and at least one data display indicator from the list of optional display indicators pointed to by the click request is obtained based on the parsing result; If a click request is detected in the filter condition setting field, the click request is parsed, and at least one data filter condition pointed to by the click request is obtained based on the parsing result. The data filter condition can be customized by the user in the filter condition customization box, or can be selected by the user in the filter condition selectable list. If a click request is detected in the association algorithm setting field, the click request is parsed, and at least one indicator association algorithm pointed to by the click request is obtained based on the parsing result. The indicator association algorithm can be customized by the user in the association algorithm customization box, or can be selected by the user in the association algorithm optional list.

6. A data report generation device, characterized in that, The data report generation device implements the steps of the data report generation method as described in any one of claims 1 to 5, wherein the data report generation device comprises: The data acquisition module is used to collect real-time and offline data to be analyzed as initial data and store it in a preset relational database. The data forwarding module is used to obtain the initial data from the relational database based on the Druid data connection pool and send it to the Hive data warehouse, wherein the Hive data warehouse adopts a distributed caching method; The interface initialization module is used to perform offline ETL processing on the initial data based on the Hive data warehouse and display it on the preset data analysis interface. The analysis instruction acquisition module is used to acquire the data analysis dimension, data display indicators, data filtering conditions, and indicator correlation algorithms selected by the user through click events on the data analysis interface. The execution statement generation module is used to generate HQL execution statements based on the data analysis dimensions, data display indicators, data filtering conditions, and indicator association algorithms. The analysis result acquisition module is used to convert the HQL execution statement into a data analysis plan and store it in the HDFS file system based on the Hadoop cluster, execute the data analysis plan, and obtain the data analysis results; The analysis results display module is used to display the data analysis report in a preset area within the data analysis interface.

7. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the data report generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data report generation method as described in any one of claims 1 to 5.

Images