Big data analysis method for realizing variable parameter control, terminal and medium

Abstract

The invention relates to the technical field of big data analysis, and discloses a big data analysis method for realizing variable parameter control, a terminal and a medium. The method comprises the following steps: firstly, constructing a big data analysis scene, and customizing and selecting various selectable parameters required by big data analysis in the scene, including indexes, time, dimensions and a visual result display mode; checking and analyzing the index, the time, the dimension and the visualization result display mode to obtain conditions required by SQL query; then, on the basis of the conditions required by the SQL query, a query structure is assembled by applying a JsqlParser tool; executing a query statement corresponding to the query structure to obtain an analysis result; and finally, carrying out visual display on the analysis result according to the visual result display mode. The method can adapt to setting of different filtering parameters and dimension conditions, variable parameter control is achieved, diversified analysis tasks can be flexibly processed, and compatibility of different big data analysis scenes is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and particularly to a big data analysis method, a terminal and a medium for realizing variable parameter control. Background Art

[0002] Data is an important asset of an enterprise. Analyzing big data can help the enterprise better understand various resources, enhance the data value, and is of extremely important significance for promoting the development of the enterprise.

[0003] However, the existing enterprise big data analysis methods are usually fixed analysis modes and scenarios, which are difficult to adapt to data sets with variable parameters and different dimensions, and cannot meet all potential analysis requirements. In dealing with complex data in a fixed analysis scenario, the analysis results may be affected by parameter selection and have low accuracy. Summary of the Invention

[0004] To solve the technical problems existing in the prior art, the present invention provides a big data analysis method, a terminal and a medium for realizing variable parameter control. The present invention can adapt to different filtering parameters and dimension condition settings, realizes variable parameter control, can flexibly process diversified analysis tasks, and improves the compatibility of different big data analysis scenarios.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention discloses a big data analysis method for realizing variable parameter control, including the following steps:

[0007] S1. Construct a big data analysis scenario, and custom-select various optional parameters required for big data analysis in this scenario, including indicators, time, dimensions, and the visualization result display mode;

[0008] S2. Check and parse the indicators, time, dimensions, and the visualization result display mode to obtain the conditions required for SQL query;

[0009] S3. Assemble the query structure by using the JsqlParser tool based on the conditions required for the SQL query;

[0010] S4. Execute the query statement corresponding to the query structure to obtain the analysis result;

[0011] S5. Visualize the analysis result according to the visualization result display mode.

[0012] As a further improvement of the above solution, step S1 includes the following specific steps:

[0013] S11. Input the custom scenario name of the current big data analysis;

[0014] S12. Determine the current scenario analysis conditions, select the current scenario analysis type, and at the same time select multiple data statistical dimensions; among them, one data type and one data statistical dimension together constitute a data indicator, and one or more optional additional filtering conditions are attached to each data indicator. The additional filtering conditions include filtering indicators and limit values of the filtering conditions. There are options of any-satisfy-and relationship or any-satisfy-or relationship among multiple additional filtering conditions, and each additional filtering condition takes effect on the data indicator it is attached to;

[0015] S13. Select the global filtering conditions; the global filtering conditions include filtering indicators and limit values of the filtering conditions; the global filtering conditions take effect on all data indicators simultaneously;

[0016] S14. Select the grouping method and sorting method; the grouping method is used to specify that data indicators are grouped and viewed according to specific attributes; the sorting method is used to specify that data is viewed in a specific order; among them, there is one or more of the grouping method and the sorting method;

[0017] S15. Select the time parameter; the time parameter is used to specify the statistical time range and statistical time dimension;

[0018] S16. Select the visualization result display method; the visualization result display method is page chart display or Excel table display.

[0019] As a further improvement of the above solution, step S2 includes the following specific steps:

[0020] S21. Perform null check and format verification, and eliminate all indicators with mandatory options being empty and indicators whose formats do not meet the set requirements;

[0021] S22. Perform Json parsing and conversion on the selected indicator parameters, obtain the conditions required for the SQL query after corresponding one by one with the received indicator parameters, and receive them with the corresponding data indicator Java object; the data indicator Java object includes: indicator, grouping method, filtering conditions, time, and visualization result display method, corresponding to the results of steps S12 to S16.

[0022] As a further improvement of the above solution, in step S3, the query structure is composed of a pre-query structure and a post-query structure; among them, the construction method of the pre-query structure includes the following steps:

[0023] S31-1. Place the two conditions of data type and time range in the bottommost query at the front, and construct the Condition structure of JsqlParser;

[0024] S31-2. Traverse the Condition structure to assemble and generate an Expression structure;

[0025] S31-3. Construct the first SelectItem array structure of JsqlParser with data metrics;

[0026] S31-4. Use the table name of the big data storage as an alias parameter;

[0027] S31-5. Construct the first-level PlainSelect structure; wherein, use the Expression structure as the where attribute of the first-level PlainSelect structure, use the first SelectItem array structure as the SelectItem attribute of the first-level PlainSelect structure, and use the alias parameter as the Alias attribute of the first-level PlainSelect structure;

[0028] S31-6. Set the alias parameter of the pre-subquery;

[0029] S31-7. Construct the SubSelect structure of JsqlParser; wherein, use the first-level PlainSelect structure as the FromItem attribute of the SubSelect structure, and use the alias parameter of the pre-subquery as the Alias attribute of the SubSelect structure;

[0030] S31-8. Construct the second SelectItem array structure of JsqlParser according to the filtering metrics in the additional filtering and global filtering;

[0031] S31-9. Construct the Table and Column structures of JsqlParser according to the limit values of the filtering conditions in the additional filtering and global filtering and the database tables where the filtering metrics are located, and encapsulate them into the first Join structure to determine the associated query logic;

[0032] S31-10. Construct the second-level PlainSelect structure; wherein, use the SubSelect structure as the FromItem attribute of the second-level PlainSelect structure, use the second SelectItem array structure as the SelectItem attribute of the second-level PlainSelect structure, and use the first Join structure as the Joins attribute of the second-level PlainSelect structure;

[0033] S31-11. Set the alias parameter of the pre-with query;

[0034] S31-12. Construct the WithItem structure, i.e., the pre-query structure; wherein, use the second-level PlainSelect structure as the withSelectBody attribute of the WithItem structure, and at the same time use the alias parameter of the pre-with query as the withName attribute of the WithItem structure.

[0035] As a further improvement of the above solution, the construction method of the post-query structure includes the following steps:

[0036] S32-1. Use the database table where the filtering metrics are located as the Table and Column structures of the post-query structure, and the two together form the second Join structure;

[0037] S32-2. Construct the third-level PlainSelect structure, i.e., the post-query structure; wherein, use the query parameters in the pre-query structure as the SelectItems attribute of the third-level PlainSelect structure, use the alias parameter of the pre-with query as the FromItem attribute of the third-level PlainSelect structure, use the second Join structure as the Joins attribute of the third-level PlainSelect structure, use the grouping method as the GroupByElement attribute of the third-level PlainSelect structure, and use the sorting method as the OrderByElement attribute of the third-level PlainSelect structure.

[0038] As a further improvement of the above solution, in step S3, assemble the query structure by using the pre-query structure as the withWithItemList attribute of the query structure and the post-query structure as the withSelectBody attribute of the query structure.

[0039] As a further improvement of the above solution, step S4 includes the following specific steps:

[0040] S41. Convert the query structure into a query statement of string type;

[0041] S42. Batch replace the specific keywords in the query statement with the keywords of the query statement adapted to Clickhouse syntax, so as to obtain the adjusted query statement;

[0042] S43. Configure the Clickhouse database connection attributes, and then perform result query based on the adjusted query statement to obtain the analysis result.

[0043] As a further improvement of the above solution, step S5 includes the following specific steps:

[0044] S51. Calculate the percentage of each row of data in the analysis result respectively, and at the same time, complete the missing data in the analysis result with default values;

[0045] S52. Determine the visualization result display method. If it is the page chart display, the data of the analysis result will be processed and rendered by the front-end page and then displayed in the form of a chart; if it is the Excel table display, call the Excel writing tool to write the data of the analysis result according to the Excel file specification and then generate an Excel file.

[0046] The present invention also discloses a computer terminal, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. It is characterized in that when the processor executes the program, the steps of a big data analysis method for implementing variable parameter control as described above are realized.

[0047] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. It is characterized in that when the program is executed by the processor, the steps of a big data analysis method for implementing variable parameter control as described above are realized.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] 1. The big data analysis method provided by the present invention can adapt to different filtering parameters and dimension conditions, realizes variable parameter control, can flexibly process diversified analysis tasks, and is compatible with the requirements of different big data analysis scenarios.

[0050] In traditional big data analysis scenarios, a fixed SQL template is mostly used, which is difficult to adapt to different business requirements, and a large amount of development work is required to change the query logic. The present invention allows users to freely select analysis indicators, dimensions, time ranges, grouping methods, etc. through variable parameter configuration, avoiding the limitations of fixed query templates. In addition, the present invention adopts additional filtering conditions and global screening conditions, supports a more refined data screening method, and improves the accuracy of data analysis. Finally, the present invention controls the data presentation method through grouping + sorting, supports complex data analysis requirements, such as multi-dimensional cross-analysis, Top-N sorting, etc.

[0051] 2. The present invention incorporates the JsqlParser technology, which simplifies the processing of SQL statements, reduces the processing and adaptation difficulties in multiple scenarios; can be compatible with various SQL language features, provides a high degree of customization options; and facilitates SQL conversion and rewriting, avoiding repeated development.

[0052] Traditional big data analysis uses the method of handwritten SQL splicing, with poor code readability, easy to generate SQL injection risks, and there are differences in SQL syntax for different databases, resulting in high migration or adaptation costs. The present invention uses JsqlParser to parse SQL statements, which can dynamically modify the query structure, such as adjusting WHERE conditions, adding JOIN relationships, etc. And it is compatible with multiple SQL dialects such as MySQL and ClickHouse, making SQL conversion more flexible and reducing the database migration cost. In addition, it supports SQL optimization, such as parsing filtering conditions in advance, removing invalid queries, and reducing the database calculation pressure. Finally, a collaborative mechanism of pre-query structure and post-query structure is adopted, splitting the SQL query into a pre-query structure (With CTE) and a post-query structure (main query), and improving the query efficiency through hierarchical optimization to avoid unnecessary calculation overhead.

[0053] 3. The present invention improves the accuracy of big data analysis results: Parameters and dimensions support custom fine-grained division, further improving the accuracy of big data analysis results. Description of the Drawings

[0054] Figure 1 It is a flowchart of the big data analysis method for realizing variable parameter control in Embodiment 1 of the present invention.

[0055] Figure 2 It is a schematic diagram of the big data analysis scenario in Embodiment 1 of the present invention.

[0056] Figure 3 It is a hierarchical schematic diagram of the first layer, the second layer and the third layer of PlainSelect in Embodiment 1 of the present invention.

[0057] Figure 4 It is a table diagram of the analysis results in Embodiment 1 of the present invention.

[0058] Figure 5 It is a bar chart of the analysis results in Embodiment 1 of the present invention.

[0059] Figure 6 It is a pie chart of the analysis results in Embodiment 1 of the present invention.

[0060] Figure 7 It is a schematic diagram of the structure of a computer terminal in Embodiment 2 of the present invention. Detailed Embodiments

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0062] Embodiment 1

[0063] Please refer to Figure 1 , this embodiment provides a big data analysis method for implementing variable parameter control, including the following steps, namely S1 to S5.

[0064] S1. Construct a big data analysis scenario, and custom-select various optional parameters required for big data analysis in this scenario, including metrics, time, dimensions, and the visualization result display method.

[0065] Step S1 includes the following specific steps, namely S11 to S16.

[0066] S11. Input the custom scenario name of the current big data analysis. As Figure 2 shown, in this embodiment, "Distribution of Daily Request Counts for Specified Interfaces in xx Computer Room on February xx, 2025" is input to analyze the distribution of daily request counts for specified interfaces in xx computer room under specified conditions in February 2025.

[0067] S12. Determine the current scenario analysis conditions, select the current scenario analysis type, and at the same time select multiple data statistical dimensions.

[0068] Among them, one data type and one data statistical dimension together constitute a data indicator, and one or more optional additional filtering conditions are attached to each data indicator. The additional filtering conditions include filtering indicators and limit values of the filtering conditions. There are options of any-satisfy-and relationship or any-satisfy-or relationship among multiple additional filtering conditions, and each additional filtering condition takes effect on the data indicator to which it is attached.

[0069] Here, matching the scenario case in step S11, for this scenario, the statistical indicator can be determined: the total number of requests for the specified interface; the indicator filtering condition: the computer room is xx computer room; the additional filtering condition: supports the "and" relationship or the "or" relationship.

[0070] S13. Select global screening conditions; the global screening conditions include filtering indicators and limit values of the filtering conditions; the global screening conditions take effect on all data indicators simultaneously.

[0071] In this embodiment, the global filtering condition: the requested url is the specified interface.

[0072] S14. Select the grouping method and sorting method; the grouping method is used to specify that data metrics are grouped and viewed according to specific attributes; the sorting method is used to specify that data is viewed in a specific order; where there is one or more of the grouping method and the sorting method.

[0073] For the above scenario, the grouping method can be determined: group and count by date; sorting method: ascending order by date.

[0074] S15. Select the time parameter; the time parameter is used to specify the statistical time range and statistical time dimension. For the above scenario, the statistical time range: February 1, 2025 to February 28, 2025; statistical time dimension: day.

[0075] S16. Select the visualization result display method; the visualization result display method is page chart display or Excel table display. In some embodiments, the display types support various forms such as bar charts, bar graphs, pie charts, tables, etc.

[0076] S2. Verify and parse the metrics, time, dimension, and visualization result display method to obtain the conditions required for the SQL query.

[0077] Step S2 includes the following specific steps:

[0078] S21. Perform null and format verification, and eliminate all metrics with mandatory options being empty and metrics with formats not meeting the set requirements.

[0079] S22. Perform Json parsing and conversion on the selected metric parameters, and obtain the conditions required for the SQL query after corresponding to the received metric parameters one by one, and receive them with the corresponding data metric Java object; the data metric Java object includes: metrics, grouping method, filtering conditions, time, and visualization result display method, corresponding to the results of steps S12 to S16.

[0080] In this embodiment, the data metric Java object is as follows:

[0081] Metrics: corresponding to the statistical metric in S12 (the number of requests to the specified interface);

[0082] Grouping method: corresponding to the grouping method in S14 (group and count by date);

[0083] Filtering conditions: corresponding to the global filtering conditions in S13, the metric filtering conditions in S12, etc.;

[0084] Time: corresponding to the statistical time range in S15;

[0085] Visualization result display method: corresponding to the supported display types in S16.

[0086] S3. Assemble the query structure using the JsqlParser tool based on the required conditions of the SQL query.

[0087] In step S3, the query structure consists of a pre-query structure and a post-query structure; among them, the construction method of the pre-query structure includes the following steps, namely S31-1 to S31-12.

[0088] S31-1. Place the two conditions of data type and time range in the bottommost query at the front to construct the Condition structure of JsqlParser. Condition is a condition that includes attribute fields, operators (equal to, not equal to, contains, does not contain, etc.), and attribute condition values.

[0089] S31-2. Traverse the Condition structure to assemble and generate the Expression structure. Expression is the base class for all expressions in SQL, covering from basic column names to complex conditional logics. Taking the time range from 2025-02-01 to 2025-02-28 as an example, its Expression structure is shown in Table 1:

[0090] Table 1. Example of Expression structure

[0091]

[0092] S31-3. Use data metrics to construct the first SelectItem (SelectItem: represents a specific item in the SELECT clause) array structure of JsqlParser.

[0093] Example: event.uid as uid

[0094] S31-4. Use the table name of the big data storage as an alias parameter.

[0095] Example: from server_log_all as event

[0096] S31-5. Construct the first-level PlainSelect (PlainSelect: describes the entire SELECT structure, including SelectItem, Expression, etc.) structure; among them, use the Expression structure as the where attribute of the first-level PlainSelect structure, use the first SelectItem array structure as the SelectItem attribute of the first-level PlainSelect structure, and use the alias parameter as the Alias attribute of the first-level PlainSelect structure.

[0097] Example SQL: select event.uid as uid,event.date as date from server_log_all event where event.url='xx interface'and event.date between'2025-02-01'and'2025-02-28'

[0098] S31-6. Set the alias parameter of the pre-subquery.

[0099] S31-7. Construct the SubSelect structure of JsqlParser; among them, use the first-level PlainSelect structure as the FromItem attribute of the SubSelect structure (FromItem: the data source of the FROM clause, supporting tables, subqueries, and joins), and use the alias parameter of the pre-subquery as the Alias attribute of the SubSelect structure;

[0100] S31-8. Construct the second SelectItem array structure of JsqlParser according to the filtering metrics in the additional filtering and global filtering.

[0101] S31-9. Construct the Table and Column structures of JsqlParser according to the limit values of the filtering conditions in the additional filtering and global filtering and the database tables where the filtering metrics are located, and encapsulate them into the first Join structure to determine the associated query logic.

[0102] S31-10. Build the second-level PlainSelect structure; among them, use the SubSelect structure as the FromItem attribute of the second-level PlainSelect structure, use the second SelectItem array structure as the SelectItem attribute of the second-level PlainSelect structure, and use the first Join structure as the Joins attribute of the second-level PlainSelect structure; as shown in the following S31-11 sql example: global inner join server_info on eventBasic.sid = server_info.id.

[0103] S31-11. Set the alias parameters for the preposed with query. The example sql is shown in Table 2:

[0104] Table 2. Example of alias parameters for the preposed with query

[0105]

[0106]

[0107] S31-12. Build the WithItem structure, which is the preposed query structure. WithItem: The WITH clause item represents a CTE (Common Table Expression), that is, a named subquery in the WITH clause. For example, in the above S31-11 example, with eventWith as (...).

[0108] Among them, use the second-level PlainSelect structure as the withSelectBody attribute of the WithItem structure (withSelectBody: represents the main body of the SELECT query in the CTE expression. For example, in the above S31-11 example, all the content except with eventWith as), and at the same time use the alias parameters of the preposed with query as the withName attribute of the WithItem structure (when building a common table expression (CTE), withName is used to define the alias of the subquery. For example, eventWith after with in the above S31-11 is the withName here).

[0109] The construction method of the postposed query structure includes the following steps, namely S32-1 to S32-2.

[0110] S32-1. Use the database table where the filtering metrics are located as the Table (data table) and Column (data column) structures of the postposed query structure, and the two together form the second Join structure; asFigure 3 Perform a global left join with user_tag_sff where eventWith.e_user_id = user_tag_sff.user_id.

[0111] S32-2. Construct the third-level PlainSelect structure, i.e., the post-query structure; specifically, use the query parameters in the pre-query structure as the SelectItems property of the third-level PlainSelect structure, use the alias parameters of the pre-with query as the FromItem property of the third-level PlainSelect structure, use the second Join structure as the Joins property of the third-level PlainSelect structure, use the grouping method as the GroupByElement property of the third-level PlainSelect structure (GroupByElement: represents the GROUP BY clause, including a list of grouping expressions, for example, group by unit in the following SQL statement means aggregating and grouping by the unit field), and use the sorting method as the OrderByElement property of the third-level PlainSelect structure (OrderByElement: represents a sorting item in the ORDER BY clause (such as column name + sorting direction), for example, order by unit asc in the following SQL statement indicates sorting in the forward direction (asc) by the unit field, and if it is desc, it indicates reverse sorting).

[0112] Examples of the levels of the first, second, and third-level PlainSelect are shown as Figure 3 follows.

[0113] In step S3, assemble the query structure by using the pre-query structure as the withWithItemList property of the query structure and the post-query structure as the withSelectBody property of the query structure. An example SQL is shown in Table 3.

[0114] Table 3. Example SQL of the query structure

[0115]

[0116]

[0117] S4. Execute the query statement corresponding to the query structure to obtain the analysis result.

[0118] Step S4 includes the following specific steps:

[0119] S41. Convert the query structure into a query statement of string type;

[0120] S42. Batch replace specific keywords in the query statement with keywords adapted to the query statement of Clickhouse syntax, so as to obtain an adjusted query statement;

[0121] S43. Configure the Clickhouse database connection property, and then perform result query based on the adjusted query statement to obtain an analysis result. The above SQL execution result is shown in Table 4.

[0122] Table 4. SQL Execution Result

[0123] unit Counts 2025-02-01 438 2025-02-02 134 2025-02-03 243 2025-02-04 387 2025-02-05 2408 2025-02-06 2016 2025-02-07 2186 2025-02-08 1819 2025-02-09 760 2025-02-10 2396 2025-02-11 2836 2025-02-12 2453 2025-02-13 2483

[0124] S5. Visualize the analysis result according to the visualization result display method.

[0125] Step S5 includes the following specific steps:

[0126] S51. Calculate the percentage of each row of data in the analysis result respectively, and at the same time fill in default values for the missing data in the analysis result;

[0127] S52. Determine the visualization result display method. If it is the page chart display, the data of the analysis result is processed and rendered by the front-end page and then displayed in the form of a chart; if it is the Excel table display, call the Excel writing tool to write the data of the analysis result according to the Excel file specification and then generate an Excel file.

[0128] As Figures 4 to 6 shown, the analysis results presented in the form of a table, a bar chart, and a pie chart respectively. In some embodiments, it may also be in other forms.

[0129] Embodiment 2

[0130] This embodiment provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the big data analysis method described in Embodiment 1 are implemented.

[0131] As Figure 7 shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to at least one processor 101. In this embodiment, the specific connection medium between the processor 101 and the memory 102 is not limited. Figure 7 It is taken as an example that the processor 101 and the memory 102 are connected through a bus 100. The bus 100 is in Figure 7The middle is represented by a thick line. The connection manners between other components are only for illustrative purposes and are not limiting. The bus 100 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 7 it is only represented by a thick line in the middle, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be called a controller, and there is no limitation on the name.

[0132] In this embodiment, the memory 102 stores instructions executable by at least one processor 101. By executing the instructions stored in the memory 102, the at least one processor 101 can execute the foregoing method.

[0133] Among them, the processor 101 is the control center of the device. It can connect various parts of the entire control device through various interfaces and lines. By running or executing the instructions stored in the memory 102 and calling the data stored in the memory 102, various functions of the device and process data, so as to monitor the device as a whole.

[0134] In a possible design, the processor 101 may include one or more processing units. The processor 101 can integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 101. In some embodiments, the processor 101 and the memory 102 can be implemented on the same chip. In some embodiments, they can also be separately implemented on independent chips.

[0135] The processor 101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the big data analysis method disclosed in conjunction with Embodiment 1 can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor 101.

[0136] The memory 102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 102 may include at least one type of storage medium, for example, it may include flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disc, and so on. The memory 102 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 102 in this embodiment may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0137] By programming the design of the processor 101, the code corresponding to the security verification method introduced in the foregoing embodiments can be solidified into the chip, so that the chip can execute Figure 7 the steps of the big data analysis method shown. How to program the design of the processor 101 is a well-known technology to those skilled in the art and will not be elaborated here.

[0138] Embodiment 3

[0139] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the big data analysis method as described in Embodiment 1 are implemented.

[0140] The computer-readable storage medium may include flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage medium may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Of course, the storage medium may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the memory is generally used to store the operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various data that have been output or will be output.

[0141] As described above, only the preferred specific embodiments of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A big data analysis method for implementing variable parameter control, characterized in that, It includes the following steps: S1. Construct a big data analysis scenario, and custom-select various optional parameters required for big data analysis in this scenario, including metrics, time, dimensions, and the visualization result display method; S2. Verify and parse the metrics, time, dimensions, and visualization result display method to obtain the conditions required for SQL query; S3. Based on the conditions required for SQL query, use the JsqlParser tool to assemble the query structure; S4. Execute the query statement corresponding to the query structure to obtain the analysis result; S5. Visualize the analysis result according to the visualization result display method.

2. The big data analysis method for implementing variable parameter control according to claim 1, wherein, Step S1 includes the following specific steps: S11. Input the custom scenario name of the current big data analysis; S12. Determine the analysis conditions of the current scenario, select the analysis type of the current scenario, and at the same time select multiple data statistical dimensions; among them, one data type and one data statistical dimension together constitute a data metric, and one or more optional additional filtering conditions are attached to each data metric. The additional filtering conditions include filtering metrics and the limit values of the filtering conditions. There are options of any-satisfy-and relationship or any-satisfy-or relationship among multiple additional filtering conditions, and each additional filtering condition takes effect on the data metric it is attached to; S13. Select the global filtering conditions; the global filtering conditions include filtering metrics and the limit values of the filtering conditions; the global filtering conditions take effect on all data metrics simultaneously; S14. Select the grouping method and sorting method; the grouping method is used to specify that data metrics are grouped and viewed according to specific attributes; the sorting method is used to specify that data is viewed in a specific order; among them, there is one or more of the grouping method and the sorting method; S15. Select the time parameter; the time parameter is used to specify the statistical time range and statistical time dimension; S16. Select the visualization result display method; the visualization result display method is page chart display or Excel table display.

3. A big data analysis method for implementing variable parameter control according to claim 2, characterized in that, Step S2 includes the following specific steps: S21. Execute null check and format verification, and eliminate the metrics with mandatory options being empty and the metrics with formats not meeting the set requirements; S22. Perform Json parsing and conversion on the selected metric parameters, and obtain the conditions required for the SQL query after corresponding to the received metric parameters one by one, and receive them with the corresponding data metric Java object; the data metric Java object includes: metrics, grouping method, filtering conditions, time, and visualization result display method, corresponding to the results of steps S12 to S16.

4. A big data analysis method for implementing variable parameter control according to claim 3, characterized in that, In step S3, the query structure consists of a pre-query structure and a post-query structure; among them, the construction method of the pre-query structure includes the following steps: S31-1. Place the two conditions of data type and time range in the bottommost query at the front to construct the Condition structure of JsqlParser; S31-2. Traverse the Condition structure to assemble and generate the Expression structure; S31-3. Construct the first SelectItem array structure of JsqlParser using data metrics; S31-4. Use the table name for big data storage as an alias parameter; S31-5. Construct the first-level PlainSelect structure; among them, use the Expression structure as the where attribute of the first-level PlainSelect structure, use the first SelectItem array structure as the SelectItem attribute of the first-level PlainSelect structure, and use the alias parameter as the Alias attribute of the first-level PlainSelect structure; S31-6. Set the alias parameter for the pre-subquery; S31-7. Construct the SubSelect structure of JsqlParser; among them, use the first-level PlainSelect structure as the FromItem attribute of the SubSelect structure, and use the alias parameter of the pre-subquery as the Alias attribute of the SubSelect structure; S31-8. Construct the second SelectItem array structure of JsqlParser according to the filtering metrics in the additional filtering and global filtering; S31-9. Construct the Table and Column structures of JsqlParser according to the limit values of the filtering conditions and the database tables where the filtering metrics are located in the additional filtering and global filtering, and encapsulate them into the first Join structure to determine the associated query logic; S31-10. Construct the second-level PlainSelect structure; among them, use the SubSelect structure as the FromItem attribute of the second-level PlainSelect structure, use the second SelectItem array structure as the SelectItem attribute of the second-level PlainSelect structure, and use the first Join structure as the Joins attribute of the second-level PlainSelect structure; S31-11. Set the alias parameter for the pre-with query; S31-12. Construct the WithItem structure, i.e., the pre-query structure; among them, use the second-level PlainSelect structure as the withSelectBody attribute of the WithItem structure, and at the same time use the alias parameter of the pre-with query as the withName attribute of the WithItem structure.

5. A big data analysis method for implementing variable parameter control according to claim 4, characterized in that, The construction method of the post-query structure includes the following steps: S32-1. Use the database table where the filtering metric is located as the Table and Column structures of the post-query structure, and the two together form the second Join structure; S32 - 2. Construct the third - level PlainSelect structure, i.e., the post - query structure; among them, use the query parameters in the pre - query structure as the SelectItems property of the third - level PlainSelect structure, use the alias parameters of the pre - with query as the FromItem property of the third - level PlainSelect structure, use the second Join structure as the Joins property of the third - level PlainSelect structure, use the grouping method as the GroupByElement property of the third - level PlainSelect structure, and use the sorting method as the OrderByElement property of the third - level PlainSelect structure.

6. A big data analysis method for implementing variable parameter control according to claim 5, characterized in that, In step S3, assemble the query structure by using the pre - query structure as the withWithItemList property of the query structure and the post - query structure as the withSelectBody property of the query structure.

7. A big data analysis method for implementing variable parameter control according to claim 2, characterized in that, Step S4 includes the following specific steps: S41. Convert the query structure into a query statement of string type. S42. Batch - replace specific keywords in the query statement with keywords of a query statement adapted to Clickhouse syntax to obtain an adjusted query statement. S43. Configure the Clickhouse database connection properties, and then perform result query based on the adjusted query statement to obtain the analysis result.

8. A big data analysis method for implementing variable parameter control according to claim 7, characterized in that Step S5 includes the following specific steps: S51. Calculate the percentage of each row of data in the analysis result respectively, and at the same time, fill in default values for the missing data in the analysis result. S52. Judge the visualization result display method. If it is the page chart display, the data of the analysis result is processed and rendered by the front - end page and then displayed in the form of a chart. If it is the Excel table display, call the Excel writing tool to write the data of the analysis result according to the Excel file specification to generate an Excel file.

9. A computer terminal, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a big - data analysis method for implementing variable - parameter control as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of a big - data analysis method for implementing variable - parameter control as described in any one of claims 1 to 8.

Images