Intelligent Index Report Calculation and Generation Method, Device and Medium

Abstract

The present application discloses a method, device and medium for calculating and generating an intelligent indicator report, mainly relating to the technical field of indicator reports, and is used to solve the problems of low efficiency, easy error, high cost of generating by commercial intelligence tools, and difficulty in function customization in the existing manual generation of indicator reports. It includes: obtaining source data corresponding to a to-be-statistical report through any computing node, and then synchronizing the target table corresponding to the source data to the analysis database of the current computing node; the current computing node determines the required subject data from the shared database according to the subject data name and query conditions, and generates the actual number of processing tasks based on the actual quantity of the required subject data and the unique identifier of the subject data; each computing node determines the required preset indicator items and obtains calculation results; notifying the computing node that generates the processing task of the calculation results; when the computing node obtains the calculation results of all processing tasks, filling them into the target table and storing the target table out of the library for preservation.

Description

Technical Field

[0001] This application relates to the technical field of report generation, and particularly to an intelligent index report calculation and generation method, device, and medium. Background Art

[0002] Due to the importance of data-driven decision-making, the demand for intelligent index calculation and report generation in various industries and fields is also increasing. Users can automatically generate sales reports in different dimensions (such as by region, product, time period) based on sales data and predict future sales according to historical trends. Existing solutions usually include Excel and other spreadsheet statistics, dedicated business tools, etc.

[0003] For some small and medium-sized enterprises, if manual calculation and report generation are used, there are usually problems of low efficiency and poor scalability. Manual calculation and report generation are time-consuming and laborious, especially when dealing with a large amount of data or complex calculations, it is easy to make mistakes. When the data volume or the types of indicators increase, the manual method is difficult to maintain and lacks automation capabilities, with poor scalability. And if dedicated business intelligence tools are used, there are also the following problems: 1) Usually expensive, especially for small and medium-sized enterprises, the initial investment and continuous usage costs are relatively high; 2) High learning cost, specialized training is required to fully utilize the functions of these tools, and non-technical personnel may have difficulty getting started; 3) Difficult to customize functions. Although business intelligence tools provide rich functions, for very specific business requirements, customization may be limited.

[0004] Therefore, there is an urgent need for an intelligent index report calculation and generation method, device, and medium to solve the problems of low efficiency, easy error in manual generation, high cost in business intelligence tool generation, and difficulty in function customization. Summary of the Invention

[0005] In view of the above deficiencies of the prior art, this application provides an intelligent index report calculation and generation method, device, and medium to solve the problems of low efficiency, easy error in manual generation of existing index reports, high cost in business intelligence tool generation, and difficulty in function customization.

[0006] In a first aspect, this application provides an intelligent index report calculation and generation method, and the method includes:

[0007] When a computing node starts up, it registers the node information on a preset Redis and periodically sends heartbeat data to Redis through a local heartbeat mechanism; through any computing node, it obtains the source data corresponding to the report to be statistically analyzed, and then synchronizes the target table corresponding to the source data to the analysis database of the current computing node; it obtains the report statistical dimension parameters, main data names, query conditions, and statistical items corresponding to the report to be statistically analyzed; among them, the statistical items include a number of preset index items, and the preset index items include index ID, index attribute, index name, and preset operation expression; it generates the current task ID and stores the corresponding relationship between the current task ID and the statistical items in a shared database; the current computing node determines the required main data from the shared database according to the main data name and query conditions, and generates a number of processing tasks equal to the actual quantity based on the actual quantity of the required main data and the unique identifier of the main data; among them, each processing task contains a unique identifier of the main data; the current computing node determines all the currently alive computing nodes through the heartbeat data on Redis, and notifies the current task ID and the processing tasks to each computing node; each computing node determines the required preset index items according to the received current task ID, and then processes the main data corresponding to the processing task using the preset index items to obtain a calculation result; it notifies the calculation result to the computing node that generates the processing task; when the computing node obtains the calculation results of all the processing tasks, it fills them into the target table and saves the target table out of the library.

[0008] Further, when a computing node starts up, it registers the node information on a preset Redis and periodically sends heartbeat data to Redis through a local heartbeat mechanism, specifically including: when a computing node starts up, it registers the node information on a preset Redis and sets an expiration time; after successful registration, all computing nodes periodically send heartbeat data to Redis through a local heartbeat mechanism every preset time period for heartbeat keep-alive.

[0009] Further, the source data includes a database and a target table name; through any computing node, it obtains the source data corresponding to the report to be statistically analyzed, and then synchronizes the target table corresponding to the source data to the analysis database of the current computing node, specifically including: through any computing node, it obtains the database and target table name corresponding to the report to be statistically analyzed; then it uses the target table name to find the corresponding target table in the database, and then synchronizes the target table to the analysis database of the current computing node.

[0010] Further, distributing the current task ID and the processing tasks to each computing node specifically includes:

[0011] Taking the modulus of the actual quantity of the processing tasks by the number of computing nodes to calculate the allocation quantity of the processing tasks that each computing node needs to calculate.

[0012] Further, the required preset indicator items are determined according to the received current task ID, and then the preset indicator items are used to process the main data corresponding to the task to obtain the calculation results, which specifically include:

[0013] After receiving the processing task, the computing node creates a task thread for each processing task and puts it into the thread pool;

[0014] Obtain the correspondence between the current task ID and the preset indicator item from the shared database, and then determine the preset indicator item required for the current task ID, and then create a calculation task for each preset indicator item and add it to the thread pool;

[0015] Generate the md5 value corresponding to each preset indicator item according to the unique identifier of the subject data, indicator ID, current task ID and report statistical dimension parameters corresponding to the preset indicator item;

[0016] Query the cached calculation result corresponding to the current md5 value from redis. When the cached calculation result exists, use it directly. When the cached calculation result does not exist, query the cached calculation result corresponding to the current md5 value from the preset indicator result correction and solidification table. When the cached calculation result exists, use it directly. When the cached calculation result does not exist, obtain the indicator ID of the current preset indicator item. Determine the sub-indicator ID corresponding to the current indicator ID, and then load the preset indicator item corresponding to the sub-indicator ID. When the preset indicator item does not have a sub-indicator ID and does not have a cached calculation result, obtain the indicator attribute corresponding to the current preset indicator item. When the indicator attribute is a non-scoring attribute, dynamically execute the preset operation expression in the current preset indicator item through the @SelectProvider annotation of mybatis to obtain the calculation result. When the indicator attribute is a scoring attribute, use the AviatorExpression method to execute the preset operation expression in the current preset indicator item to obtain the calculation result.

[0017] Furthermore, after obtaining the calculation result, the method further includes:

[0018] The calculation result, md5 value, and indicator name are cached in redis as the assembly result.

[0019] Furthermore, the method further comprises:

[0020] When it is determined that the cached calculation result corresponding to any md5 value in the shared database is an incorrect result, the data corresponding to the md5 value is deleted from the shared database; the correct calculation result corresponding to the md5 value is obtained, and then the correct calculation result corresponding to the md5 value is stored in the preset indicator result correction solidification table.

[0021] Furthermore, the current task ID and processing task are notified to each computing node, including:

[0022] Notify each computing node of the current task ID and the processing task through the Redis message queue.

[0023] Notify the computing node that generates the processing task of the calculation result, specifically including:

[0024] Write the calculation result into the shared database and notify the computing node that generates the processing task.

[0025] In a second aspect, the present application provides an intelligent metric report calculation and generation device, which includes:

[0026] A processor;

[0027] And a memory, on which executable code is stored. When the executable code is executed, the processor is caused to execute an intelligent metric report calculation and generation method as described in any one of the above.

[0028] In a third aspect, the present application provides a non-volatile computer storage medium, on which computer instructions are stored. When the computer instructions are executed, an intelligent metric report calculation and generation method as described in any one of the above is implemented.

[0029] Those skilled in the art can understand that the present application has at least the following beneficial effects:

[0030] The present application can encapsulate the calculation tasks of the report metrics according to the subject information into a task list by any computing node that receives the report calculation request, and allocate the tasks to the service nodes that have been registered and are alive in Redis for calculation, realizing decentralized distributed calculation, which is beneficial to the scaling of the computing host and makes full use of the existing hardware resources. In addition, using each computing node for metric calculation improves concurrency while enhancing the calculation efficiency. In summary, the present application realizes the automatic generation of the target table, without the need for manual generation or the aid of business intelligence tools, solving the problems of low efficiency, easy errors in manual generation of existing metric reports, and high costs in generating reports using business intelligence (BI) tools.

[0031] In addition, the present application can determine the statistical items according to the requirements, and then determine the specific preset operation expressions, thereby realizing the function of expression customization, and further solving the problems such as difficult function customization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The following describes some embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0033] Figure 1 is a flowchart of an intelligent metric report calculation and generation method provided by an embodiment of the present application.

[0034] Figure 2It is a schematic diagram of the internal structure of an intelligent index report calculation and generation device provided by an embodiment of the present application. Detailed implementation manners

[0035] Those skilled in the art should understand that the embodiments described below are only the preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through these preferred embodiments. These preferred embodiments are only used to explain the technical principles of the present disclosure and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts should still fall within the protection scope of the present disclosure.

[0036] It should also be noted that the term "including", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0037] The technical solutions proposed in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0038] An embodiment of the present application provides an intelligent index report calculation and generation method. As Figure 1 shown, the method provided by the embodiment of the present application mainly includes the following steps:

[0039] Step 110: When the calculation node starts, register the node information on the preset redis, and regularly send heartbeat data to redis through the local heartbeat mechanism.

[0040] It should be noted that the calculation node can be a server. Redis can be a Redis server supported by a preset central platform.

[0041] This step can be specifically: when the calculation node starts, register the node information on the preset redis and set the expiration time; after successful registration, all calculation nodes regularly send heartbeat data to redis through the local heartbeat mechanism at preset time intervals for heartbeat keep-alive.

[0042] It should be noted that the expiration time can be 30 seconds, such as report:node1 ip:port 30s; the heartbeat data can be set to redis regularly through the local heartbeat mechanism (key:report:node1 value:ip:port ttl:30S), and the preset time period can be 20 seconds.

[0043] Step 120: Obtain the source data corresponding to the report to be statistically analyzed through any computing node, and then synchronize the target table corresponding to the source data to the analysis database of the current computing node.

[0044] It should be noted that the source data includes the database and the target table name.

[0045] This application can be specifically as follows:

[0046] Through any computing node, obtain the database and target table name corresponding to the report to be statistically analyzed; then use the target table name to find the corresponding target table in the database, and then synchronize the target table to the analysis database of the current computing node.

[0047] Step 130: Obtain the report statistical dimension parameters, main data name, query conditions, and statistical items corresponding to the report to be statistically analyzed; generate the current task ID and store the corresponding relationship between the current task ID and the statistical items in the shared database.

[0048] It should be noted that the statistical items include several preset index items, and the preset index items include index ID, index attribute, index name, and preset operation expression.

[0049] Among them, the preset operation expression can be an arithmetic formula or a judgment logic.

[0050] The judgment logic can be:

[0051] Total score 10, maximum 10, minimum 3 points;

[0052] The score expression of the product customer evaluation is as follows:

[0053] if (monthly positive review rate <= 50%) {total score - 7};

[0054] elsif (monthly positive review rate > 50% && monthly positive review rate <= 70%) {total score - 5};

[0055] elsif (monthly positive review rate > 70% && monthly positive review rate <= 90%) {total score - 3};

[0056] elsif (monthly positive review rate > 90% && monthly positive review rate < 100%) {total score - 1};

[0057] else{total score}

[0058] Meanwhile, it is specified that the maximum total score will not exceed 10 and the minimum will not be lower than 3.

[0059] Step 140: The current computing node determines the required subject data from the shared database according to the subject data name and query conditions, and generates the actual number of processing tasks based on the actual quantity of the required subject data and the unique identifier of the subject data.

[0060] It should be noted that each processing task contains a unique identifier of the subject data for easy distinction.

[0061] Step 150: The current computing node determines all the currently alive computing nodes through the heartbeat data on redis, and notifies the current task ID and the processing tasks to each computing node; each computing node determines the required preset metric items according to the received current task ID, and then processes the subject data corresponding to the processing tasks by using the preset metric items to obtain the calculation results; the calculation results are notified to the computing node that generates the processing tasks.

[0062] As an example, notifying the current task ID and the processing tasks to each computing node can specifically be:

[0063] Notifying the current task ID and the processing tasks to each computing node through the redis message queue.

[0064] Among them, distributing the current task ID and the processing tasks to each computing node can specifically be:

[0065] Taking the remainder of the actual quantity of the processing tasks divided by the number of computing nodes to calculate the allocation quantity of the processing tasks that each computing node needs to calculate.

[0066] Among them, determining the required preset metric items according to the received current task ID, and then processing the subject data corresponding to the processing tasks by using the preset metric items to obtain the calculation results can specifically be:

[0067] S1: After receiving the processing tasks, the computing node creates a task thread for each processing task and puts it into the thread pool.

[0068] S2: Obtain the correspondence between the current task ID and the preset metric items from the shared database, and then determine the preset metric items required by the current task ID, and then create a calculation task for each preset metric item and add it to the thread pool.

[0069] It should be noted that when each computing node calculates indicators, thread pools are created for processing tasks and indicator calculation tasks respectively. At the same time, the indicator information calculated by each computing node is cached in Redis later. While improving concurrency, the calculated results that have been completed are fully utilized, thereby improving the calculation efficiency.

[0070] S3. Generate the md5 value corresponding to each preset indicator item according to the unique identifier of the main data corresponding to the preset indicator item, the indicator ID, the current task ID, and the report statistical dimension parameter.

[0071] It should be added that the md5 value can be calculated by the prior art in S3, and the calculation process of this application is not limited.

[0072] S4. Query the cached calculation result corresponding to the current md5 value from Redis. When there is a cached calculation result, use it directly; when there is no cached calculation result, query the cached calculation result corresponding to the current md5 value from the preset indicator result correction and solidification table. When there is a cached calculation result, use it directly; when there is no cached calculation result, obtain the indicator ID of the current preset indicator item; determine the sub-indicator ID corresponding to the current indicator ID, and then load the preset indicator item corresponding to the sub-indicator ID; when the preset indicator item has no sub-indicator ID and no cached calculation result, obtain the indicator attribute corresponding to the current preset indicator item; when the indicator attribute is a non-scoring attribute, dynamically execute the preset operation expression in the current preset indicator item through the @SelectProvider annotation of MyBatis to obtain the calculation result; when the indicator attribute is a scoring attribute, use the AviatorExpression method to execute the preset operation expression in the current preset indicator item to obtain the calculation result.

[0073] It should be added that the sub-indicator ID is a kind of indicator ID in this application. The indicator ID itself can also be the sub-indicator ID of other indicator IDs. In addition, the corresponding relationship between this indicator ID and the sub-indicator ID is preset by those skilled in the art according to actual needs. In addition, the preset operation expression (mainly logic) of the above scoring attribute cannot be dynamically executed through the @SelectProvider annotation of MyBatis. In this application, it is limited here to use the AviatorExpression method to execute the preset operation expression in the current preset indicator item to obtain the calculation result (AviatorExpression is a high-performance Java expression evaluation engine based on bytecode, focusing on providing fast expression calculation capabilities. It is mainly used for dynamically calculating expressions, especially in scenarios where frequent calculations are required). Here, Code Aviator is used for indicator assembly and calculation logic processing, and its dynamic expression ability is used to define the integral rule and calculate the integral indicator, making the user operation more intuitive.

[0074] In addition, for the convenience of subsequent direct use and to reduce computational consumption, after obtaining the calculation result, the method further includes: caching the calculation result, the md5 value, and the metric name as an assembled result in Redis.

[0075] In addition, the present application is manually corrected and solidified, and the solidification result of the current metric can participate in the calculation of integral metrics or other composite metrics. Among them, the method of correction and solidification can be:

[0076] When it is determined that the cached calculation result corresponding to any md5 value in the shared database is an incorrect result, the data corresponding to the md5 value is deleted from the shared database; the correct calculation result corresponding to the md5 value is obtained, and then the correct calculation result corresponding to the md5 value is stored in a preset metric result correction and solidification table.

[0077] As an example, the calculation node that generates the processing task is notified of the calculation result. Specifically, it can be: writing the calculation result into the shared database and notifying the calculation node that generates the processing task.

[0078] Based on the above steps, those skilled in the art can know that the present application assigns tasks to service nodes that are registered and alive in Redis for calculation, realizing decentralized distributed calculation, which is beneficial to the scaling of computing hosts and makes full use of existing hardware resources.

[0079] Step 160, when the calculation node obtains the calculation results of all processing tasks, fill them into the target table and save the target table out of the library.

[0080] The above is the method embodiment in the present application. Based on the same inventive concept, the embodiment of the present application also provides an intelligent metric report calculation and generation device. As Figure 2 shown, the device includes: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes an intelligent metric report calculation and generation method as in the above embodiment.

[0081] Specifically, when the computing node starts up, the server registers the node information on the preset Redis and regularly sends heartbeat data to Redis through the local heartbeat mechanism; obtains the source data corresponding to the report to be counted through any computing node, and then synchronizes the target table corresponding to the source data to the analysis database of the current computing node; obtains the report statistical dimension parameters, main data names, query conditions, and statistical items corresponding to the report to be counted; among them, the statistical items include several preset index items, and the preset index items include index ID, index attribute, index name, and preset operation expression; generates the current task ID and stores the corresponding relationship between the current task ID and the statistical items in the shared database; the current computing node determines the required main data from the shared database according to the main data name and query conditions, and generates the actual number of processing tasks based on the actual quantity of the required main data and the unique identifier of the main data; among them, each processing task includes a unique identifier of the main data; the current computing node determines all the currently alive computing nodes through the heartbeat data on Redis, and notifies the current task ID and the processing tasks to each computing node; each computing node determines the required preset index items according to the received current task ID, and then processes the main data corresponding to the processing task by using the preset index items to obtain the calculation result; notifies the calculation result to the calculation node that generates the processing task; when the calculation node obtains the calculation results of all the processing tasks, fills them into the target table, and saves the target table out of the library.

[0082] In addition, the embodiment of the present application also provides a non-volatile computer storage medium, on which executable instructions are stored, and when the executable instructions are executed, the above-mentioned intelligent index report calculation generation method is implemented.

[0083] So far, the technical solutions of the present disclosure have been described in combination with multiple previous embodiments. However, it is easy for those skilled in the art to understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principle of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. An intelligent index report calculation and generation method, characterized in that The method includes: When a computing node starts, register the node information on a preset Redis, and regularly send heartbeat data to Redis through the local heartbeat mechanism; Through any computing node, obtain the source data corresponding to the report to be statistically reported, and then synchronize the target table corresponding to the source data to the analysis database of the current computing node; Obtain the report statistical dimension parameters, main data names, query conditions, and statistical items corresponding to the report to be statistically reported; among them, the statistical items include several preset index items, and the preset index items include index ID, index attribute, index name, and preset operation expression; generate the current task ID and store the corresponding relationship between the current task ID and the statistical items in the shared database; The current computing node determines the required main data from the shared database according to the main data name and query conditions, and generates the actual number of processing tasks based on the actual quantity of the required main data and the unique identifier of the main data; each processing task includes a unique identifier of the main data; The current computing node determines all the currently alive computing nodes through the heartbeat data on Redis, and notifies the current task ID and the processing tasks to each computing node; each computing node determines the required preset index items according to the received current task ID, and then uses the preset index items to process the main data corresponding to the processing tasks to obtain the calculation results; specifically including: After receiving the processing task, the computing node creates a task thread for each processing task and puts it into the thread pool; obtains the corresponding relationship between the current task ID and the preset index items from the shared database, and then determines the preset index items required by the current task ID, and then creates a calculation task for each preset index item and adds it to the thread pool; generates the md5 value corresponding to each preset index item according to the unique identifier of the main data corresponding to the preset index item, index ID, current task ID, and report statistical dimension parameters; queries the cached calculation result corresponding to the current md5 value from Redis, and directly uses it when there is a cached calculation result; when there is no cached calculation result, queries the cached calculation result corresponding to the current md5 value from the preset index result correction and solidification table, and directly uses it when there is a cached calculation result; when there is no cached calculation result, obtains the index ID of the current preset index item; determines the sub-index ID corresponding to the current index ID, and then loads the preset index item corresponding to the sub-index ID; when the preset index item does not have a sub-index ID and there is no cached calculation result, obtains the index attribute corresponding to the current preset index item; when the index attribute is a non-scoring attribute, dynamically execute the preset operation expression in the current preset index item through the @SelectProvider annotation of MyBatis to obtain the calculation result; when the index attribute is a scoring attribute, use the AviatorExpression method to execute the preset operation expression in the current preset index item to obtain the calculation result; Among them, notifying the current task ID and the processing tasks to each computing node specifically includes: Notify each computing node of the current task ID and processing tasks through the Redis message queue; Notify the computing node that generated the processing task of the calculation result; specifically including: writing the calculation result into the shared database and notifying the computing node that generated the processing task; When the computing node obtains the calculation results of all processing tasks, fill them into the target table and export and save the target table.

2. The method for calculating and generating an intelligent metric report according to claim 1, wherein When the computing node starts, register the node information on the preset Redis and send heartbeat data to Redis regularly through the local heartbeat mechanism, specifically including: When the computing node starts, register the node information on the preset Redis and set the expiration time; After successful registration, all computing nodes send heartbeat data to Redis regularly through the local heartbeat mechanism at preset time intervals for heartbeat keep-alive.

3. The method for calculating and generating an intelligent metric report according to claim 1, wherein The source data includes the database and the target table name; Through any computing node, obtain the source data corresponding to the report to be statistically analyzed, and then synchronize the target table corresponding to the source data to the analysis database of the current computing node, specifically including: Through any computing node, obtain the database and target table name corresponding to the report to be statistically analyzed; then use the target table name to find the corresponding target table in the database, and then synchronize the target table to the analysis database of the current computing node.

4. The method for calculating and generating an intelligent metric report according to claim 1, wherein Distribute the current task ID and processing tasks to each computing node, specifically including: Take the modulus of the actual number of processing tasks by the number of computing nodes to calculate the allocation quantity of the processing tasks that each computing node needs to calculate.

5. The method for calculating and generating an intelligent metric report according to claim 1, wherein, After obtaining the calculation result, the method further includes: Cache the calculation result, md5 value, and metric name as the assembled result in Redis.

6. The method for calculating and generating an intelligent index report according to claim 1, wherein The method further includes: When it is determined that the cached calculation result corresponding to any md5 value in the shared database is an incorrect result, delete the data corresponding to the md5 value from the shared database; Obtain the correct calculation result corresponding to the md5 value, and then store the correct calculation result corresponding to the md5 value in the preset metric result correction and solidification table.

7. An intelligent index report calculation and generation device, characterized in that, The device includes: A processor; And a memory, on which executable code is stored. When the executable code is executed, the processor executes an intelligent metric report calculation and generation method according to any one of claims 1-6.

8. A non-volatile computer storage medium, characterized in that, Computer instructions are stored thereon, and when the computer instructions are executed, an intelligent metric report calculation and generation method according to any one of claims 1-6 is implemented.

Images