Data index self-calculation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a data index self-calculation method, device and equipment and a storage medium, and the method comprises the steps: according to a data index system, obtaining a blood relationship including a logic relationship and a derivation relationship between data indexes; constructing a data index relation tree according to the blood relationship among the data indexes; the data index relation tree is monitored, and when data of a certain atomic index in the data index relation tree changes, a broadcast message used for notifying the related data indexes having the blood relationship with the atomic index with the data change to be updated and calculated is pushed; and in response to the broadcast message, performing updating calculation on the related data indexes in sequence according to the blood relationship by taking the atomic index with the data change as a starting point. According to the method, automatic and intelligent calculation of data indexes and timely response and processing of data changes are realized, and the calculation efficiency and accuracy of data tasks are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data indicator self-calculation method, device, equipment and storage medium. Background Art

[0002] In the traditional data computing task processing process, computing tasks are often scheduled according to manually configured task scheduling rules to make them proceed in an orderly manner to improve the efficiency of computing resource utilization and system performance. However, the data indicator system involved in many data computing tasks contains a large number of atomic indicators, derived indicators, and composite indicators, and there are logical and derived relationships between each indicator, which makes the design of task scheduling rules more difficult and greatly reduces the efficiency and accuracy of big data computing tasks.

[0003] There is a method for distributed indicator calculation, which first assembles a workflow by parsing the indicator formula and parses the corresponding dependencies of the workflow, then distributes and executes the calculation tasks according to the task type based on the workflow dependencies, and finally verifies the indicator lineage relationship and calculation results recorded in the workflow based on the calculation results to generate the target indicator.

[0004] However, the above method cannot respond and process in a timely manner when the indicator data changes, and has the defects of low calculation efficiency and accuracy. Summary of the invention

[0005] In order to overcome the defects of low efficiency and accuracy of data index calculation in the prior art, the present invention proposes the following technical solutions:

[0006] In a first aspect, the present invention provides a data indicator self-calculation method, comprising:

[0007] According to the data indicator system, the blood relationship between the data indicators is obtained; the blood relationship includes the logical relationship and the derived relationship between the data indicators;

[0008] Constructing a data indicator relationship tree according to the blood relationship between the data indicators;

[0009] Monitoring the data indicator relationship tree, and when data of a certain atomic indicator in the data indicator relationship tree changes, pushing a broadcast message for notifying to update and calculate related data indicators that have a blood relationship with the atomic indicator with the changed data;

[0010] In response to the broadcast message, starting from the atomic indicator where the data has changed, the relevant data indicators are updated and calculated in sequence according to the blood relationship.

[0011] In a second aspect, the present invention further provides a data indicator self-calculation method device, comprising:

[0012] The acquisition module is used to acquire the blood relationship between data indicators according to the data indicator system. The blood relationship includes the logical relationship and the derived relationship between the data indicators.

[0013] The construction module is used to construct a data indicator relationship tree according to the blood relationship between the data indicators.

[0014] The monitoring module is used to monitor the data indicator relationship tree, and when the data of an atomic indicator in the data indicator relationship tree changes, push a broadcast message for notifying the update calculation of related data indicators that have a blood relationship with the atomic indicator with data changes.

[0015] The calculation module is used to respond to the broadcast message, take the atomic indicator where the data change occurs as the starting point, and update and calculate the relevant data indicators in sequence according to the blood relationship.

[0016] In a third aspect, the present invention further proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the operations performed by the data indicator self-calculation method as described in any one of the schemes in the first aspect are implemented.

[0017] In a fourth aspect, the present invention further proposes a computer-readable storage medium having a program stored thereon, wherein the program is executed by a processor to perform the operations performed by the data indicator self-calculation method as described in any of the schemes in the first aspect.

[0018] The beneficial effects of the present invention include at least:

[0019] (1) The present invention constructs a data indicator relationship tree based on the blood relationship between data indicators to perform self-calculation of data indicators, thereby realizing the automated and intelligent calculation of data indicators without the need for cumbersome task scheduling configuration, thereby improving the calculation efficiency and accuracy of data tasks.

[0020] (2) The present invention monitors the data indicator relationship tree. When the data of a certain atomic indicator changes, the relevant data indicators that have a blood relationship with the atomic indicator with the data change are updated and calculated, thereby achieving timely response and processing of data changes and further improving the calculation efficiency and accuracy of data tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a data indicator self-calculation method provided in an embodiment of the present application.

[0022] Figure 2 This is a schematic diagram of the principle architecture of constructing a parallel tree in an embodiment of the present application.

[0023] Figure 3 This is a schematic diagram of the principle architecture in which the calculation logic of the indicator nodes in the embodiment of the present application is merged into the first-level indicator nodes.

[0024] Figure 4 This is a flow chart of updating and calculating relevant data indicators in an embodiment of the present application.

[0025] Figure 5 This is an architecture diagram of a data indicator self-calculation device provided in an embodiment of the present application.

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0029] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0030] Specifically, Figure 1 A flowchart of a data indicator self-calculation method provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the data indicator self-calculation method includes the following steps:

[0032] S10: According to the data indicator system, the blood relationship between the data indicators is obtained; the blood relationship includes the logical relationship and the derived relationship between the data indicators.

[0033] It is understandable that the data indicator system refers to a group of data indicators organized according to a certain hierarchical structure and classification standards, which are used to reflect the characteristics, laws and values ​​of data. The logical relationship between the data indicators refers to the mathematical or statistical operation relationship between the data indicators, such as sum, average, proportion, etc. The derivative relationship between the data indicators refers to the causal or influence relationship between the data indicators, such as a certain data indicator is the source or result of another data indicator, or a certain data indicator has a positive or negative impact on another data indicator, etc.

[0034] S20: Constructing a data indicator relationship tree according to the blood relationship between the data indicators.

[0035] Optionally, in one embodiment of the present application, S20 specifically includes the following steps:

[0036] S201: Determine the primary indicators from the data indicator system.

[0037] S202: Taking each first-level indicator as a root node, traverse other data indicators in the data indicator system, and construct a plurality of data indicator relationship trees according to the blood relationship between each first-level indicator and other data indicators.

[0038] In this embodiment, Figure 2 As shown, after constructing several data indicator relationship trees, different data indicator relationship trees with the same indicator nodes are merged with the same indicator nodes as the connection points; the merged data indicator relationship trees are pruned so that each indicator node appears only once in all data indicator relationship trees; the pruned data indicator relationship trees are split with the first-level indicator as the starting indicator node to obtain several parallel trees that can be executed in parallel.

[0039] In this embodiment, Figure 3 As shown, after obtaining several parallel trees that can be executed in parallel, in each parallel tree, a depth-first search algorithm is used to merge the calculation logic of the indicator nodes that come from the same data source and have the same data type as the first-level indicator node into the first-level indicator node.

[0040] In the specific implementation process, the depth-first search algorithm refers to starting from the first-level indicator node, searching downward along the edge of the data indicator relationship tree, and when encountering an indicator node that comes from the same data source and is of the same data type as the first-level indicator node, merging the calculation logic of the indicator node into the first-level indicator node, and deleting the indicator node from the data indicator relationship tree until all edges connected to the first-level indicator node are traversed. Then, starting from the next layer of the first-level indicator node, the above process is repeated until the entire data indicator relationship tree is traversed.

[0041] It can be understood that merging the calculation logic of the indicator nodes that come from the same data source and have the same data type as the first-level indicator node into the first-level indicator node simplifies and optimizes the data indicator relationship tree and reduces redundancy and repeated calculations in the data indicator relationship tree.

[0042] Optionally, in one embodiment of the present application, after constructing the data indicator relationship tree, the calculation logic of the indicator node in the data indicator relationship tree is parsed, and the atomic indicator or derived indicator used by the current parsing indicator node is obtained from the calculation logic, and the obtained atomic indicator or derived indicator is set as the next-level child node of the current parsing indicator node; this step is repeated until all non-atomic indicators in the data indicator relationship tree can be traced back to one or more atomic indicators.

[0043] It can be understood that by setting the obtained atomic indicator or derived indicator as the next-level child node of the current parsed indicator node, until all non-atomic indicators in the data indicator relationship tree can be traced back to one or more atomic indicators, a complete and accurate display of the blood relationship between data indicators is achieved, which facilitates the analysis and tracing of the source and impact of the data indicators, realizes the improvement and refinement of the data indicator relationship tree, improves the readability and usability of the data indicator relationship tree, provides a basis and support for subsequent data indicator self-calculation, reduces the complexity and difficulty of data indicator self-calculation, and improves the efficiency and accuracy of data indicator self-calculation.

[0044] Optionally, in one embodiment of the present application, after constructing the data indicator relationship tree, atomic indicators are identified from the data indicator relationship tree and set as monitoring objects; and corresponding monitoring methods are set according to the data source type used by the atomic indicators.

[0045] In the specific implementation process, the atomic indicator refers to the most basic data indicator in the data indicator system, which is independent of the calculation results of other data indicators and is directly obtained or calculated from the data source. The data source types include static data sources and dynamic data sources. The static data source refers to a data source whose data does not change over time, such as historical data, fixed parameters, etc. The dynamic data source refers to a data source whose data changes over time, such as real-time data, variable parameters, etc. The monitoring method includes timed monitoring and event monitoring, such as mysql can monitor binlog. The timed monitoring refers to monitoring the atomic indicator at a preset time interval or time point to detect whether the data of the atomic indicator changes. The event monitoring refers to monitoring the atomic indicator according to a preset trigger condition or rule to detect whether the data of the atomic indicator meets the trigger condition or rule.

[0046] S30: monitoring the data indicator relationship tree, and when the data of an atomic indicator in the data indicator relationship tree changes, pushing a broadcast message for notifying to update and calculate related data indicators that have a blood relationship with the atomic indicator with the changed data.

[0047] Optionally, in one embodiment of the present application, when the number of data changes of an atomic indicator in the data indicator relationship tree reaches a preset change window number, a broadcast message is pushed to notify the update calculation of related data indicators that have a blood relationship with the atomic indicator where the data change occurs.

[0048] For example, atomic indicators A and B are identified, and atomic indicators A and B are set as monitoring objects. According to the data source type used by atomic indicators A and B, the corresponding monitoring method is set. Assuming that both atomic indicators A and B use dynamic data sources, set the event monitoring method. Assuming that the preset number of change windows is 5, when the number of data changes of atomic indicator A or B reaches 5, a broadcast message is pushed to notify the update calculation of related data indicators that have a blood relationship with the atomic indicator where the data change occurs.

[0049] When the number of data changes of atomic indicator A reaches 5, a broadcast message is pushed to notify derived indicators C and D to perform update calculations. In response to the broadcast message, starting from atomic indicator A, derived indicators C and D are updated and calculated in sequence according to the blood relationship. That is, the latest value of derived indicator C is calculated based on the latest data of atomic indicators A and B, and then the latest value of derived indicator D is calculated based on the latest data of derived indicator C and atomic indicator A.

[0050] S40: In response to the broadcast message, starting from the atomic indicator where the data has changed, the relevant data indicators are updated and calculated in sequence according to the blood relationship.

[0051] Optionally, in one embodiment of the present application, the broadcast message includes a broadcast sequence number, a timestamp and a data source.

[0052] In this embodiment, Figure 4 As shown, the specific steps of S40 include:

[0053] S401: Match the data source of the atomic indicator with the data change currently with the data source in the broadcast message. If the match succeeds, execute S402; if the match fails, end;

[0054] S402: Detect whether the data indicator relationship tree where the atomic indicator with data change currently exists is in execution state. If yes, add the atomic indicator with data change currently to the waiting queue and execute S403. If no, directly execute S404.

[0055] S403: Detect whether there are other atomic indicators in the data indicator relationship tree where the atomic indicator with data change currently exists and have been added to the waiting queue. If so, remove the atomic indicator with data change currently from the waiting queue and terminate the execution. If not, execute S404.

[0056] S404: According to the blood relationship, the calculation logic of each indicator node in the data indicator relationship tree where the atomic indicator with the current data change is located is executed in sequence, and the calculation result data is stored in the database;

[0057] S405: After all indicator nodes of the current data indicator relationship tree are executed, check whether there is an atomic indicator in the waiting queue. If so, repeat S402-S405 for the first atomic indicator in the waiting queue. If not, end.

[0058] Secondly, the data indicator self-calculation device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0059] Figure 5 4 is an architecture diagram of a data indicator self-calculation device according to an embodiment of the present application.

[0060] like Figure 5 As shown, the recommendation device includes: an acquisition module 100, a construction module 200, a monitoring module 300 and a calculation module 400.

[0061] Among them, the acquisition module 100 is used to obtain the blood relationship between data indicators according to the data indicator system; the blood relationship includes the logical relationship and the derived relationship between the data indicators; the construction module 200 is used to construct a data indicator relationship tree according to the blood relationship between the data indicators; the monitoring module 300 is used to monitor the data indicator relationship tree, and when the data of an atomic indicator in the data indicator relationship tree changes, push a broadcast message for notifying the update calculation of related data indicators that have a blood relationship with the atomic indicator with data changes; the calculation module 400 is used to respond to the broadcast message, starting from the atomic indicator with data changes, and updating and calculating the related data indicators in turn according to the blood relationship.

[0062] It should be noted that the above explanation of the data indicator self-calculation method embodiment is also applicable to the data indicator self-calculation device of this embodiment, and will not be repeated here.

[0063] Figure 6 The computer device 500 provided in the embodiment of the present application is a schematic diagram of the structure of the computer device 500. The computer device 500 includes: a memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor 502.

[0064] When the processor 502 executes the program, the data indicator self-calculation method provided in the above embodiment is implemented.

[0065] Furthermore, the computer device 500 also includes: a communication interface 503 for communication between the memory 501 and the processor 502 .

[0066] The memory 501 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0067] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0068] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0069] The processor 502 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0070] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned data indicator self-calculation method.

[0071] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0072] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0073] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0074] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0075] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0076] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A data indicator self-calculation method, characterized in that: include: According to the data indicator system, the blood relationship between the data indicators is obtained; the blood relationship includes the logical relationship and the derived relationship between the data indicators; Constructing a data indicator relationship tree according to the blood relationship between the data indicators; Monitoring the data indicator relationship tree, and when data of a certain atomic indicator in the data indicator relationship tree changes, pushing a broadcast message for notifying to update and calculate related data indicators that have a blood relationship with the atomic indicator with the changed data; In response to the broadcast message, starting from the atomic indicator where the data has changed, the relevant data indicators are updated and calculated in sequence according to the blood relationship.

2. The data indicator self-calculation method according to claim 1, characterized in that: After constructing the data indicator relationship tree, the method further includes: Parse the calculation logic of the indicator node in the data indicator relationship tree, and obtain the atomic indicator or derived indicator used by the current parsing indicator node from the calculation logic, and set the obtained atomic indicator or derived indicator as the next-level child node of the current parsing indicator node; repeat this step until all non-atomic indicators in the data indicator relationship tree can be traced back to one or more atomic indicators.

3. The data indicator self-calculation method according to claim 1, characterized in that: According to the blood relationship between the data indicators, a data indicator relationship tree is constructed, which specifically includes: Determine the first-level indicators from the data indicator system; Taking each first-level indicator as the root node, traverse other data indicators in the data indicator system, and build several data indicator relationship trees based on the blood relationship between each first-level indicator and other data indicators.

4. The data indicator self-calculation method according to claim 3, characterized in that: After constructing a plurality of data indicator relationship trees, the method further includes: For different data indicator relationship trees with the same indicator node, the same indicator node is used as the connection point for merging; Prune the merged data indicator relationship tree so that each indicator node appears only once in all data indicator relationship trees; Taking the first-level indicator as the starting indicator node, the pruned data indicator relationship tree is split to obtain several parallel trees that can be executed in parallel.

5. The data indicator self-calculation method according to claim 4, characterized in that: After obtaining a plurality of parallel trees that can be executed in parallel, the method further includes: In each parallel tree, a depth-first search algorithm is used to merge the calculation logic of the indicator nodes that come from the same data source and have the same data type as the first-level indicator node into the first-level indicator node.

6. The data indicator self-calculation method according to claim 1, characterized in that: After constructing the data indicator relationship tree, the method further includes: Identify atomic indicators from the data indicator relationship tree, and set the atomic indicators as monitoring objects; Set the corresponding monitoring mode according to the data source type used by the atomic indicator.

7. The data indicator self-calculation method according to claim 1, characterized in that: When the data of an atomic indicator in the data indicator relationship tree changes, a broadcast message is pushed to notify the update calculation of the related data indicators that have a blood relationship with the atomic indicator with the data change, specifically including: When the number of data changes of a certain atomic indicator in the data indicator relationship tree reaches the preset change window number, a broadcast message is pushed to notify the update calculation of related data indicators that have a blood relationship with the atomic indicator where the data change occurs.

8. The data indicator self-calculation method according to claim 1, characterized in that: The broadcast message includes a broadcast sequence number, a timestamp and a data source.

9. The data indicator self-calculation method according to claim 8, characterized in that: In response to the broadcast message, starting from the atomic indicator where the data has changed, the relevant data indicators are updated and calculated in sequence according to the blood relationship. The specific steps include: Step a: Match the data source of the atomic indicator with the data change with the data source in the broadcast message. If the match is successful, execute step b. If the match fails, end. Step b: Detect whether the data indicator relationship tree where the atomic indicator with data change currently exists is in the execution state. If so, add the atomic indicator with data change currently to the waiting queue and execute step c. If not, directly execute step d. Step c: Check whether there are other atomic indicators in the data indicator relationship tree where the atomic indicator with data change currently exists and have been added to the waiting queue. If so, remove the atomic indicator with data change currently from the waiting queue and terminate the execution. If not, execute step d. Step d: According to the blood relationship, the calculation logic of each indicator node in the data indicator relationship tree where the atomic indicator with the current data change is located is executed in sequence, and the calculation result data is stored in the database; Step e: After all indicator nodes of the current data indicator relationship tree are executed, check whether there is an atomic indicator in the waiting queue. If so, repeat steps b-e for the first atomic indicator in the waiting queue. If not, end.

10. A data index self-calculation device, characterized in that: include: The acquisition module is used to obtain the blood relationship between data indicators based on the data indicator system; The blood relationship includes the logical relationship and the derived relationship between the data indicators; A construction module, used to construct a data indicator relationship tree according to the blood relationship between the data indicators; A monitoring module, used for monitoring the data indicator relationship tree, and when the data of an atomic indicator in the data indicator relationship tree changes, pushing a broadcast message for notifying the update calculation of the related data indicators having a blood relationship with the atomic indicator with the changed data; The calculation module is used to respond to the broadcast message, take the atomic indicator where the data change occurs as the starting point, and update and calculate the relevant data indicators in sequence according to the blood relationship.

11. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the operations performed by the data indicator self-calculation method as described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and the program is used by a processor to execute the operations performed by the data indicator self-calculation method as described in any one of claims 1 to 9.

Images