Data processing method and device, computer equipment and readable storage medium
By obtaining the pending data and incremental data in the preset data warehouse, determining the hierarchical data model and performing aggregation calculation, the inefficiency problem caused by changes in the calculation rules in the existing technology is solved, and efficient update and processing of the data warehouse is achieved.
Patent Information
- Application Number
- CN202310662423.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
Existing data processing methods require rewriting programs when calculation rules change, resulting in low computational efficiency and inability to efficiently process data updates.
By obtaining the pending data and incremental data in the preset data warehouse, determining the hierarchical data model, and using aggregation rules to calculate the aggregated data, automatically update the data warehouse, and improving data processing efficiency.
Automatic aggregation processing based on incremental data is realized, which improves the efficiency of data processing and reduces the need to recalculate all data.
Smart Images

Figure CN119088809A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of data warehouses, and in particular to a data processing method, apparatus, computer equipment, and readable storage medium. Background Art
[0002] In the construction of the budget management system, there are many scenarios of data aggregation. For example, the detailed data entered by the user is converted into data of concern to finance according to the configuration rules.
[0003] The existing method is usually to write a corresponding program according to the rules to perform calculations and output the results. However, when the calculation rules change, the existing data processing method often needs to rewrite the program for calculation, and all the data needs to be calculated again for each calculation, which has low calculation efficiency. Summary of the invention
[0004] The embodiments of the present application provide a data processing method, apparatus, computer device and readable storage medium, aiming to solve the problem of low computing efficiency in existing data processing methods.
[0005] On the one hand, an embodiment of the present application provides a data processing method, including:
[0006] Acquire the data to be processed, and determine the incremental data between the pre-stored data in the preset data warehouse and the data to be processed;
[0007] Determine, according to the configuration rules corresponding to each data model in the preset data warehouse, a hierarchical data model in the data model corresponding to the data to be processed;
[0008] Determining aggregated data corresponding to the hierarchical data model according to the incremental data and an aggregation rule corresponding to the hierarchical data model;
[0009] The data model in the preset data warehouse is updated according to the aggregated data to obtain an updated data warehouse.
[0010] As a feasible embodiment of the present application, determining the hierarchical data model corresponding to the data to be processed in the data model according to the configuration rules corresponding to each data model in the preset data warehouse includes:
[0011] Based on the data source report corresponding to the data to be processed and the report configuration rules corresponding to each data model in the preset data warehouse, determine the first-level data model in the data model corresponding to the data source report;
[0012] According to the data flow rules corresponding to each data model in the preset data warehouse, a second-level data model having a data flow association relationship with the first-level data model is screened out from the preset data warehouse;
[0013] The first hierarchical data model and the second hierarchical data model are determined as hierarchical data models corresponding to the data to be processed.
[0014] As a feasible embodiment of the present application, before the step of screening out a second-level data model having a data flow association relationship with the first-level data model from the preset data warehouse according to the data flow rules corresponding to each data model in the preset data warehouse, the method further includes:
[0015] Providing target data flow rules corresponding to the target hierarchical data model in the preset data warehouse on a preset interface;
[0016] In response to an update configuration instruction for the target data flow rule, generating an updated data flow rule;
[0017] The updated data flow rule is associated with the target level data model and stored in the preset data warehouse.
[0018] As a feasible embodiment of the present application, determining the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model includes:
[0019] Calculate first aggregated data corresponding to the first hierarchical data model according to the incremental data and a first aggregation rule corresponding to a first hierarchical data model in the hierarchical data model;
[0020] According to the first aggregated data and the second aggregation rule corresponding to the second hierarchical data model in the hierarchical data model, the second aggregated data corresponding to the second hierarchical data model is calculated; wherein corresponding data flow rules exist between the first hierarchical data model and the second hierarchical data model.
[0021] As a feasible embodiment of the present application, determining the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model includes:
[0022] Obtaining the update frequency corresponding to the data to be processed;
[0023] If the update frequency is higher than a preset frequency threshold, then obtaining associated data corresponding to the incremental data;
[0024] Aggregate data corresponding to the hierarchical data model is determined according to the incremental data, the associated data, and an aggregation rule corresponding to the hierarchical data model.
[0025] As a feasible embodiment of the present application, before the step of determining the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model, the method further includes:
[0026] According to the data volume of the incremental data, determining a target node corresponding to the data volume from preset computing nodes;
[0027] Calculation is performed in the target node according to the incremental data and the aggregation rule corresponding to the hierarchical data model to obtain the aggregated data corresponding to the hierarchical data model.
[0028] As a feasible embodiment of the present application, after the step of updating the data model in the preset data warehouse according to the aggregated data to obtain an updated data warehouse, the method further includes:
[0029] In response to a query instruction input by a user, based on a data request carried in the query instruction, target data corresponding to the data request is read from a data model in the preset data warehouse;
[0030] Based on the data type corresponding to the target data, the target data is displayed in a preset layout corresponding to the data type.
[0031] On the other hand, an embodiment of the present application further provides a data processing device, including:
[0032] An acquisition module is used to acquire data to be processed and determine incremental data between pre-stored data in a preset data warehouse and the data to be processed;
[0033] A determination module, configured to determine a hierarchical data model corresponding to the data to be processed in the hierarchical data model according to configuration rules corresponding to each hierarchical data model in the preset data warehouse;
[0034] an aggregation module, configured to determine the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model;
[0035] An updating module is used to update the data model in the preset data warehouse according to the aggregated data to obtain an updated data warehouse.
[0036] On the other hand, an embodiment of the present application further provides a computer device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned data processing method.
[0037] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the above-mentioned data processing method.
[0038] The data processing method provided by the present application calculates the incremental data between the pre-stored data and the data to be processed after obtaining the data to be processed, then finds the hierarchical data model corresponding to the data to be processed from the data model stored in the data warehouse, and uses the aggregation rules corresponding to the hierarchical data model to calculate the aggregated data of each model to complete the update of the data model in the data warehouse and obtain the updated data warehouse. The data processing method provided by the embodiment of the present application can automatically complete the aggregation processing of data based on the incremental data, thereby improving the processing efficiency of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present invention;
[0041] Figure 1 (a) is a schematic diagram of an implementation scenario of a data processing method provided in an embodiment of the present application
[0042] Figure 2 A schematic diagram of a process flow of filtering out a hierarchical data model corresponding to data to be processed provided in an embodiment of the present application;
[0043] Figure 3 A schematic diagram of a step flow for configuring data flow rules provided in an embodiment of the present application;
[0044] Figure 4 A schematic diagram of a step flow of calculating aggregated data provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram of a flow chart of steps for updating incremental data in batches based on an update frequency provided in an embodiment of the present application;
[0046] Figure 6 A schematic diagram of a step flow of performing aggregate calculation based on a given node provided in an embodiment of the present application;
[0047] Figure 7 A schematic diagram of a step flow of displaying data provided in an embodiment of the present application;
[0048] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0049] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of the present invention.
[0051] In the embodiments of the present application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in the embodiments of the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present invention obscure. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the embodiments of the present application.
[0052] To facilitate understanding of the data processing method provided by the embodiment of the present application, the implementation scenario diagram of the method is first described. The data processing method provided by the embodiment of the present application is mainly used for the construction of a data management system. For example, in a common financial budget management system, budget compilation personnel often need to enter detailed compilation data in a database, and these detailed compilation data often need to be converted into data that finance needs to be concerned about, such as cash flow data, according to configured statistical rules. In the scenario where such data need to be statistically aggregated with each other, the current data warehouse provides users with a solution to write corresponding programs based on actual needs for calculations and display the calculation results. However, when the calculation rules change, it is necessary to rewrite the corresponding program, and then recalculate all the data, resulting in the existing low efficiency of data processing.
[0053] In order to solve the above problems, the embodiment of the present application provides an implementation scheme by constructing a report model and designing the flow rules between data. Specifically, the data processing method provided in the embodiment of the present application is deployed in the form of a program on a data processing device, and the data processing device is installed in a computer device in the form of a processor. The data processing device in the data processing device executes the data processing method provided in the embodiment of the present application by running the program corresponding to the data processing method. Specifically, Figure 1 As shown, Figure 1 A schematic diagram of a data processing method provided in an embodiment of the present application. The data processing method in the embodiment of the present application includes steps 101 to 104:
[0054] 101, obtaining data to be processed, and determining incremental data between pre-stored data in a preset data warehouse and the data to be processed.
[0055] In the embodiment of the present application, in combination with the implementation scheme provided above, it can be known that the data to be processed usually refers to the data input by the user to be processed. For example, taking the implementation scenario provided above as an example, the data to be processed can be the detailed compilation data entered by the budget compilation personnel. Specifically, these detailed compilation data usually exist in the form of data models in the data warehouse. Therefore, the data processing device can calculate the difference between the pre-stored data in the data warehouse and the data to be processed, that is, the incremental data.
[0056] Further, to facilitate understanding of the implementation scenario of the data processing method provided in the embodiment of the present application, as Figure 1 As shown in (a), a schematic diagram of an implementation scenario of a data processing method is provided. It can be seen that a number of data models related to reports displayed to users are stored in the data warehouse, and corresponding data flow rules exist between these data models.
[0057] 102. Determine, according to the configuration rules corresponding to the hierarchical data models in the preset data warehouse, the hierarchical data model corresponding to the data to be processed in the hierarchical data model.
[0058] In the embodiment of the present application, based on the above, since the data warehouse stores data models associated with different hierarchical relationships, and each data model corresponds to a corresponding data configuration rule, based on the configuration rule, the hierarchical data model that has an associated relationship with the data to be processed can be screened out from the data model. Specifically, these hierarchical data models include both data models directly associated with the source report of the data to be processed and associated data models obtained by processing the data to be processed. For specific implementation schemes, please refer to the subsequent Figure 2 and its explanatory contents.
[0059] 103. Determine the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model.
[0060] In the embodiment of the present application, after the hierarchical data model corresponding to the data to be processed is determined, the aggregation rules corresponding to the hierarchical data model are further used to further aggregate the obtained incremental data to obtain the aggregated data corresponding to the hierarchical data model. Specifically, considering that there is a hierarchical relationship between the hierarchical data models that is in a sequential and progressive manner, therefore, as a feasible embodiment of the present application, after determining the aggregated data corresponding to each hierarchical data model, the aggregated data corresponding to each hierarchical data model is calculated in sequence based on the data flow rules between the hierarchical data models. For the specific implementation scheme, please refer to the subsequent Figure 4 and its explanatory contents.
[0061] Furthermore, considering that the data update frequencies in different reports are different, before aggregating the hierarchical data model, different incremental data can be processed uniformly based on the data update frequency to improve the calculation efficiency. For specific implementation solutions, please refer to the subsequent Figure 5 and its explanatory contents.
[0062] In addition, considering that the data processing method provided in the embodiment of the present application is usually run in a data warehouse, in order to ensure that the data warehouse can be dynamically expanded according to performance requirements, multi-node computing can usually be set in the process of calculating aggregated data. Therefore, as another feasible embodiment of the present application, in the process of calculating aggregated data, the appropriate computing node for calculation can be selected based on the data volume of the enhanced data. Specifically, the implementation scheme can be referred to in the subsequent Figure 6 and its explanatory contents.
[0063] 104. Update the data model in the preset data warehouse according to the aggregated data to obtain an updated data warehouse.
[0064] In the embodiment of the present application, after the above calculations obtain the aggregated data corresponding to the data models at each level, the data model in the data warehouse is further updated based on the aggregated data to obtain an updated data warehouse. At this time, in response to the query command input by the user, the data warehouse can read the corresponding data from the data stored in the data model and return it to the user. The specific implementation scheme can be referred to in the subsequent Figure 7 and its explanatory contents.
[0065] The data processing method provided by the present application calculates the incremental data between the pre-stored data and the data to be processed after obtaining the data to be processed, then finds the hierarchical data model corresponding to the data to be processed from the data model stored in the data warehouse, and uses the aggregation rules corresponding to the hierarchical data model to calculate the aggregated data of each model to complete the update of the data model in the data warehouse and obtain the updated data warehouse. The data processing method provided by the embodiment of the present application can automatically complete the aggregation processing of data based on the incremental data, thereby improving the processing efficiency of data.
[0066] like Figure 2 As shown, Figure 2 A schematic flow chart of a step of screening out a hierarchical data model corresponding to the data to be processed provided in an embodiment of the present application, specifically, comprising steps 201 to 203:
[0067] 201. Based on the data source report corresponding to the data to be processed and the report configuration rules corresponding to each data model in the preset data warehouse, determine the first-level data model in the data model corresponding to the data source report.
[0068] In the present application embodiment, combined with the above Figure 1 As can be seen from the schematic diagram of the implementation scenario of the data processing method (a), the data model in the data warehouse has a one-to-one correspondence with the user-oriented data report. Therefore, after obtaining the data to be processed, the data processing device can obtain the data source report corresponding to the data to be processed, and then determine the first-level data model in the data model corresponding to the data source report based on the report configuration rules corresponding to each data model in the data warehouse, that is, the one-to-one correspondence with the user-oriented data report. For example, Figure 1 Taking the implementation scenario diagram provided in (a) as an example, when the data source report corresponding to the data to be processed is the compilation report BR-C1, at this time, it can be determined that the first-level data model corresponding to the data source report is the compilation model MA-3.
[0069] 202. According to the data flow rules corresponding to the data models in the preset data warehouse, select the second-level data model having a data flow association relationship with the first-level data model from the preset data warehouse.
[0070] In an embodiment of the present application, in addition to the aforementioned report configuration rules including the one-to-one correspondence between the data model and the user-oriented data report, the configuration rules also include data flow rules between the data models. Specifically, after filtering out the first-level data model corresponding to the data source report from the data warehouse, the data processing device may further filter out the second-level data model that has a data flow association relationship with the first-level data model from the data warehouse. It should be noted that since the data models stored in the data warehouse may have multiple levels of hierarchical relationships, the filtered second-level data model may also include multiple levels. For example, the second-level data model includes both data models that have a direct data flow association relationship with the first-level data model and data models that have an indirect data flow association relationship with the first-level data model. For example, taking the aforementioned Figure 1 Taking the implementation scenario diagram provided in (a) as an example, when the data source report corresponding to the data to be processed is the compilation report BR-C1, at this time, it can be determined that the first-level data model corresponding to the data source report is the compilation model MA-3. At this time, the second-level data model includes both the data model MB-2 that has a direct data flow association relationship with the first-level data model and the data model MC-1 that has an indirect data flow association relationship with the first-level data model. The data model that has an indirect data flow association relationship with the first-level data model can be obtained by searching upward in sequence according to the data flow rules between data models.
[0071] Of course, as an optional embodiment of the present application, the data flow rules between data models can be determined based on actual needs, and can be updated and adjusted at any time based on the actual needs of users. For specific implementation schemes, please refer to the subsequent Figure 3 and its explanatory contents.
[0072] 203: Determine the first hierarchical data model and the second hierarchical data model as hierarchical data models corresponding to the data to be processed.
[0073] In the embodiment of the present application, after the first-level data model and the second-level data model corresponding to the data to be processed are screened out based on the configuration rules corresponding to the data model, the first-level data model and the second-level data model will be recorded as the hierarchical data model corresponding to the data to be processed for further use in subsequent aggregation calculations. For details, please refer to the subsequent Figure 4 The process of providing the aggregate calculation.
[0074] like Figure 3 As shown, Figure 3 A schematic flow chart of steps for configuring data flow rules provided in an embodiment of the present application, specifically, including steps 301 to 303:
[0075] 301 , providing target data flow rules corresponding to the target hierarchical data model in the preset data warehouse on a preset interface.
[0076] In the embodiment of the present application, the data flow rules corresponding to each data model in the data warehouse are displayed in a visual manner on a preset interface provided by the data processing device to facilitate the user to perform human-computer interaction operations.
[0077] 302 : In response to an update configuration instruction for the target data flow rule, generate an updated data flow rule.
[0078] In the embodiment of the present application, the user can directly input an update configuration instruction for the target data flow rule on the preset interface, and in response to the update configuration instruction, the data processing device can generate an update data flow rule.
[0079] 303 , associate the updated data flow rule with the target hierarchical data model and store them in the preset data warehouse.
[0080] In an embodiment of the present application, the data processing device will associate the updated data flow rules with the target hierarchical data model and store them in the data warehouse to facilitate subsequent screening of the data model.
[0081] like Figure 4 As shown, Figure 4 A schematic diagram of a step flow of calculating aggregate data provided in an embodiment of the present application, specifically, including steps 401 to 402:
[0082] 401. Calculate first aggregated data corresponding to the first hierarchical data model according to the incremental data and a first aggregation rule corresponding to the first hierarchical data model in the hierarchical data model.
[0083] In an embodiment of the present application, based on the difference between the data to be processed and the pre-stored data in the data model, that is, the incremental data, and the first aggregation rule corresponding to the first-level data model in the hierarchical data model, the adjusted data corresponding to the first-level data model under the incremental data, that is, the first aggregation data, can be calculated.
[0084] 402 . Calculate second aggregate data corresponding to the second hierarchical data model according to the first aggregate data and a second aggregation rule corresponding to the second hierarchical data model in the hierarchical data model.
[0085] In an embodiment of the present application, there are corresponding data flow rules between the first-level data model and the second-level data model. Therefore, after obtaining the adjustment data corresponding to the first-level data model, that is, the first aggregated data, the adjustment data corresponding to the second-level data model, that is, the second aggregated data, will be calculated again based on the first aggregated data and the second aggregation rules corresponding to the second-level data model.
[0086] like Figure 5 As shown, Figure 5 A schematic flow chart of a step of updating incremental data in batches based on an update frequency is provided in an embodiment of the present application, specifically, including steps 501 to 503:
[0087] 501, obtaining the update frequency corresponding to the data to be processed.
[0088] In the embodiment of the present application, the update frequency of the data to be processed generally refers to the frequency at which the data in the corresponding data report is updated. Specifically, the frequency can be calculated by the number of updates occurring within a preset period.
[0089] 502. If the update frequency is higher than a preset frequency threshold, obtain associated data corresponding to the incremental data.
[0090] When the update frequency is higher than the preset frequency threshold, it can be considered that the data in the data report is frequently updated. Therefore, in order to improve the efficiency of data processing, the data to be processed can be processed in batches, that is, the related data corresponding to the incremental data can be obtained for subsequent processing. Specifically, the related data refers to other data entered by the user in the data report within the preset period that needs to be incrementally updated.
[0091] 503 , determining the aggregated data corresponding to the hierarchical data model according to the incremental data, the associated data, and the aggregation rule corresponding to the hierarchical data model.
[0092] In the embodiment of the present application, similar to the above, after obtaining other related data that need to be processed together with the incremental data, the data processing device will count the incremental data and the related data, and then use the aggregation rules corresponding to the hierarchical data model to determine the aggregated data corresponding to the hierarchical data model. Of course, the specific scheme for obtaining the aggregated data can refer to the above steps. Figure 4 , that is, determining in sequence the first aggregate data corresponding to the first hierarchical data model and the second aggregate data corresponding to the second hierarchical data model in the hierarchical data model.
[0093] like Figure 6 As shown, Figure 6A schematic flow chart of a step of performing aggregate calculation based on a given node provided in an embodiment of the present application, specifically, including steps 601 to 602:
[0094] 601. According to the data volume of the incremental data, determine a target node corresponding to the data volume from preset computing nodes.
[0095] In the embodiment of the present application, combined with the relevant instructions provided above, it can be known that in order to ensure that the data warehouse can be dynamically expanded according to performance requirements, usually, the data warehouse sets multiple nodes to perform aggregate data calculations. Therefore, it is necessary to determine the nodes that perform aggregate calculations during the actual calculation process. Specifically, as a feasible embodiment of the present application, the data processing device can determine the target node that matches the size of the data volume from the preset computing nodes based on the data volume of the incremental data. Specifically, the match here is generally that the performance or number of the node matches the data volume of the incremental data. For example, the higher the data volume, the data processing device will filter out nodes with higher computing performance or more number from the computing nodes for calculation. Conversely, the smaller the data volume, the data processing device will filter out nodes with lower computing performance or fewer number from the computing nodes for calculation.
[0096] 602. Perform calculations in the target node according to the incremental data and the aggregation rules corresponding to the hierarchical data model to obtain aggregated data corresponding to the hierarchical data model.
[0097] After determining the target node for calculation as mentioned above, the data processing device completes the calculation in the corresponding target node according to the incremental data and the aggregation rules corresponding to the hierarchical data model, and obtains the aggregated data corresponding to the hierarchical data model for updating the data in the data warehouse.
[0098] like Figure 7 As shown, Figure 7 A schematic diagram of a step flow of displaying data provided in an embodiment of the present application, specifically, including steps 701 to 702:
[0099] 701. In response to a query instruction input by a user, target data corresponding to the data request is read from a data model in the preset data warehouse based on a data request carried in the query instruction.
[0100] In an embodiment of the present application, after completing the update of the data, the user reads the target data corresponding to the data request from the data model in the preset data warehouse based on the data request carried by the query instruction by inputting the query instruction.
[0101] 702 : Based on the data type corresponding to the target data, display the target data in a preset layout corresponding to the data type.
[0102] In an embodiment of the present application, when the target data needs to be displayed to the user, the data processing device will display the target data in a preset layout corresponding to the data type based on the data type corresponding to the target data, thereby improving the visualization of the data.
[0103] In order to better implement the data processing method provided in the embodiment of the present application, based on the data processing method, the embodiment of the present application also provides a data processing device. Figure 8 As shown, Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Specifically, the data processing device includes:
[0104] The acquisition module 801 is used to acquire the data to be processed and determine the incremental data between the pre-stored data in the preset data warehouse and the data to be processed;
[0105] A determination module 802 is used to determine a hierarchical data model corresponding to the data to be processed in the data model according to the configuration rules corresponding to each data model in the preset data warehouse;
[0106] Aggregation module 803, used to determine aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model;
[0107] The updating module 804 is used to update the data model in the preset data warehouse according to the aggregated data to obtain an updated data warehouse.
[0108] As a feasible embodiment of the present application, the determination module 802 is also used to determine the first-level data model in the data model corresponding to the data source report based on the data source report corresponding to the data to be processed and the report configuration rules corresponding to each data model in the preset data warehouse; according to the data flow rules corresponding to each data model in the preset data warehouse, the second-level data model having a data flow association relationship with the first-level data model is screened out from the preset data warehouse; and the first-level data model and the second-level data model are determined as the hierarchical data models corresponding to the data to be processed.
[0109] As a feasible embodiment of the present application, the determination module 802 is also used to provide target data flow rules corresponding to the target hierarchical data model in the preset data warehouse on a preset interface; generate updated data flow rules in response to update configuration instructions for the target data flow rules; and associate the updated data flow rules with the target hierarchical data model and store them in the preset data warehouse.
[0110] As a feasible embodiment of the present application, the aggregation module 803 is also used to calculate the first aggregated data corresponding to the first hierarchical data model according to the incremental data and the first aggregation rule corresponding to the first hierarchical data model in the hierarchical data model; calculate the second aggregated data corresponding to the second hierarchical data model according to the first aggregated data and the second aggregation rule corresponding to the second hierarchical data model in the hierarchical data model; wherein there are corresponding data flow rules between the first hierarchical data model and the second hierarchical data model.
[0111] As a feasible embodiment of the present application, the aggregation module 803 is also used to obtain the update frequency corresponding to the data to be processed; if the update frequency is higher than a preset frequency threshold, the associated data corresponding to the incremental data is obtained; according to the incremental data, the associated data and the aggregation rules corresponding to the hierarchical data model, the aggregated data corresponding to the hierarchical data model is determined.
[0112] As a feasible embodiment of the present application, the aggregation module 803 is also used to determine the target node corresponding to the data volume from the preset computing nodes according to the data volume of the incremental data; calculate in the target node according to the aggregation rules corresponding to the incremental data and the hierarchical data model to obtain the aggregated data corresponding to the hierarchical data model.
[0113] As a feasible embodiment of the present application, the update module 804 is also used to respond to a query instruction input by a user, and read the target data corresponding to the data request from the data model in the preset data warehouse based on the data request carried by the query instruction; based on the data type corresponding to the target data, display the target data in a preset layout corresponding to the data type.
[0114] For the specific definition of the data processing device, please refer to the definition of the data processing method above, which will not be repeated here. Each module in the above data processing device can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. Specifically, Fig. 9 As shown, Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.
[0115] Specifically, the computer device may include one or more processing cores of the processor 901, one or more storage media of the memory 902, a power supply 903, and an input unit 904. Those skilled in the art will understand that Fig. 9The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0116] The processor 901 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 902 and calling data stored in the memory 902, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 901 may include one or more processing cores; preferably, the processor 901 may integrate a data processor and a modem processor, wherein the data processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 901.
[0117] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data processing by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 with access to the memory 902.
[0118] The computer device also includes a power supply 903 for supplying power to various components. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 903 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0119] The computer device may further include an input unit 904, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0120] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 901 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 901 will run the application programs stored in the memory 902, thereby implementing the following steps:
[0121] Acquire the data to be processed, and determine the incremental data between the pre-stored data in the preset data warehouse and the data to be processed;
[0122] Determine, according to the configuration rules corresponding to each data model in the preset data warehouse, a hierarchical data model in the data model corresponding to the data to be processed;
[0123] Determining aggregated data corresponding to the hierarchical data model according to the incremental data and an aggregation rule corresponding to the hierarchical data model;
[0124] The data model in the preset data warehouse is updated according to the aggregated data to obtain an updated data warehouse.
[0125] Furthermore, the processor running the application stored in the memory can also implement the following steps: based on the data source report corresponding to the data to be processed and the report configuration rules corresponding to each data model in the preset data warehouse, determine the first-level data model in the data model corresponding to the data source report; according to the data flow rules corresponding to each data model in the preset data warehouse, filter out the second-level data model having a data flow association relationship with the first-level data model from the preset data warehouse; determine the first-level data model and the second-level data model as the hierarchical data models corresponding to the data to be processed.
[0126] Furthermore, the processor running the application stored in the memory can also implement the following steps: providing target data flow rules corresponding to the target hierarchical data model in the preset data warehouse on a preset interface; generating updated data flow rules in response to update configuration instructions for the target data flow rules; and associating the updated data flow rules with the target hierarchical data model and storing them in the preset data warehouse.
[0127] Furthermore, the processor running the application stored in the memory can also implement the following steps: calculating the first aggregate data corresponding to the first hierarchical data model based on the incremental data and the first aggregation rule corresponding to the first hierarchical data model in the hierarchical data model; calculating the second aggregate data corresponding to the second hierarchical data model based on the first aggregate data and the second aggregation rule corresponding to the second hierarchical data model in the hierarchical data model; wherein there are corresponding data flow rules between the first hierarchical data model and the second hierarchical data model.
[0128] Furthermore, the processor running the application stored in the memory can also implement the following steps: obtaining the update frequency corresponding to the data to be processed; if the update frequency is higher than a preset frequency threshold, obtaining the associated data corresponding to the incremental data; and determining the aggregated data corresponding to the hierarchical data model based on the incremental data, the associated data and the aggregation rules corresponding to the hierarchical data model.
[0129] Furthermore, the processor running the application stored in the memory can also implement the following steps: based on the data volume of the incremental data, determining the target node corresponding to the data volume from the preset computing nodes; performing calculations within the target node based on the incremental data and the aggregation rules corresponding to the hierarchical data model to obtain aggregated data corresponding to the hierarchical data model.
[0130] Furthermore, the processor running the application stored in the memory can also implement the following steps: in response to a query instruction input by the user, reading the target data corresponding to the data request from the data model in the preset data warehouse based on the data request carried by the query instruction; and displaying the target data in a preset layout corresponding to the data type based on the data type corresponding to the target data.
[0131] To this end, an embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the following steps are implemented:
[0132] Acquire the data to be processed, and determine the incremental data between the pre-stored data in the preset data warehouse and the data to be processed;
[0133] Determine, according to the configuration rules corresponding to each data model in the preset data warehouse, a hierarchical data model in the data model corresponding to the data to be processed;
[0134] Determining aggregated data corresponding to the hierarchical data model according to the incremental data and an aggregation rule corresponding to the hierarchical data model;
[0135] The data model in the preset data warehouse is updated according to the aggregated data to obtain an updated data warehouse.
[0136] Furthermore, when the computer program is executed by the processor, the following steps can also be implemented: based on the data source report corresponding to the data to be processed and the report configuration rules corresponding to each data model in the preset data warehouse, determine the first-level data model in the data model corresponding to the data source report; according to the data flow rules corresponding to each data model in the preset data warehouse, filter out the second-level data model that has a data flow association relationship with the first-level data model from the preset data warehouse; determine the first-level data model and the second-level data model as the hierarchical data models corresponding to the data to be processed.
[0137] Furthermore, when the computer program is executed by the processor, the following steps can also be implemented: providing target data flow rules corresponding to the target hierarchical data model in the preset data warehouse on a preset interface; generating updated data flow rules in response to update configuration instructions for the target data flow rules; and associating the updated data flow rules with the target hierarchical data model and storing them in the preset data warehouse.
[0138] Furthermore, when the computer program is executed by the processor, the following steps can also be implemented: based on the incremental data and the first aggregation rule corresponding to the first hierarchical data model in the hierarchical data model, the first aggregate data corresponding to the first hierarchical data model is calculated; based on the first aggregate data and the second aggregation rule corresponding to the second hierarchical data model in the hierarchical data model, the second aggregate data corresponding to the second hierarchical data model is calculated; wherein, there are corresponding data flow rules between the first hierarchical data model and the second hierarchical data model.
[0139] Furthermore, when the computer program is executed by the processor, the following steps can be implemented: obtaining an update frequency corresponding to the data to be processed; if the update frequency is higher than a preset frequency threshold, obtaining associated data corresponding to the incremental data; and determining the aggregated data corresponding to the hierarchical data model based on the incremental data, the associated data and the aggregation rules corresponding to the hierarchical data model.
[0140] Furthermore, when the computer program is executed by the processor, the following steps can also be implemented: based on the data volume of the incremental data, determining the target node corresponding to the data volume from the preset computing nodes; performing calculations within the target node based on the incremental data and the aggregation rules corresponding to the hierarchical data model to obtain aggregated data corresponding to the hierarchical data model.
[0141] Furthermore, when the computer program is executed by the processor, the following steps can also be implemented: in response to a query instruction input by a user, reading target data corresponding to the data request from the data model in the preset data warehouse based on the data request carried by the query instruction; and displaying the target data in a preset layout corresponding to the data type based on the data type corresponding to the target data.
[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.
[0143] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.
[0144] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0145] The above is a detailed introduction to a data processing method provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A data processing method, characterized in that: include: Acquire the data to be processed, and determine the incremental data between the pre-stored data in the preset data warehouse and the data to be processed; Determine, according to the configuration rules corresponding to each data model in the preset data warehouse, a hierarchical data model in the data model corresponding to the data to be processed; Determining aggregated data corresponding to the hierarchical data model according to the incremental data and an aggregation rule corresponding to the hierarchical data model; The data model in the preset data warehouse is updated according to the aggregated data to obtain an updated data warehouse.
2. The data processing method according to claim 1, characterized in that: The step of determining, according to the configuration rules corresponding to each data model in the preset data warehouse, a hierarchical data model corresponding to the data to be processed in the data model comprises: Based on the data source report corresponding to the data to be processed and the report configuration rules corresponding to each data model in the preset data warehouse, determine the first-level data model in the data model corresponding to the data source report; According to the data flow rules corresponding to each data model in the preset data warehouse, a second-level data model having a data flow association relationship with the first-level data model is screened out from the preset data warehouse; The first hierarchical data model and the second hierarchical data model are determined as hierarchical data models corresponding to the data to be processed.
3. The data processing method according to claim 2, characterized in that: Before the step of selecting, from the preset data warehouse, a second-level data model having a data flow association relationship with the first-level data model according to the data flow rules corresponding to each data model in the preset data warehouse, the method further includes: Providing target data flow rules corresponding to the target hierarchical data model in the preset data warehouse on a preset interface; In response to an update configuration instruction for the target data flow rule, generating an updated data flow rule; The updated data flow rule is associated with the target level data model and stored in the preset data warehouse.
4. The data processing method according to claim 1, characterized in that: The determining, according to the incremental data and the aggregation rule corresponding to the hierarchical data model, the aggregated data corresponding to the hierarchical data model includes: Calculate first aggregated data corresponding to the first hierarchical data model according to the incremental data and a first aggregation rule corresponding to a first hierarchical data model in the hierarchical data model; According to the first aggregated data and the second aggregation rule corresponding to the second hierarchical data model in the hierarchical data model, the second aggregated data corresponding to the second hierarchical data model is calculated; wherein corresponding data flow rules exist between the first hierarchical data model and the second hierarchical data model.
5. The data processing method according to claim 1, characterized in that: The determining, according to the incremental data and the aggregation rule corresponding to the hierarchical data model, the aggregated data corresponding to the hierarchical data model includes: Obtaining the update frequency corresponding to the data to be processed; If the update frequency is higher than a preset frequency threshold, then obtaining associated data corresponding to the incremental data; Aggregate data corresponding to the hierarchical data model is determined according to the incremental data, the associated data, and an aggregation rule corresponding to the hierarchical data model.
6. The data processing method according to claim 1, characterized in that: Before the step of determining the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model, the method further includes: According to the data volume of the incremental data, determining a target node corresponding to the data volume from preset computing nodes; Calculation is performed in the target node according to the incremental data and the aggregation rule corresponding to the hierarchical data model to obtain the aggregated data corresponding to the hierarchical data model.
7. The data processing method according to any one of claims 1 to 6, characterized in that: After the step of updating the data model in the preset data warehouse according to the aggregated data to obtain an updated data warehouse, the method further includes: In response to a query instruction input by a user, based on a data request carried in the query instruction, target data corresponding to the data request is read from a data model in the preset data warehouse; Based on the data type corresponding to the target data, the target data is displayed in a preset layout corresponding to the data type.
8. A data processing device, characterized in that: include: An acquisition module is used to acquire data to be processed and determine incremental data between pre-stored data in a preset data warehouse and the data to be processed; A determination module, configured to determine a hierarchical data model in the data model corresponding to the data to be processed according to configuration rules corresponding to each data model in the preset data warehouse; an aggregation module, configured to determine the aggregated data corresponding to the hierarchical data model according to the incremental data and the aggregation rule corresponding to the hierarchical data model; An updating module is used to update the data model in the preset data warehouse according to the aggregated data to obtain an updated data warehouse.
9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the data processing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Incremental data acquisition method and device, incremental data sending method and device and computer storage medium
CN111930768A
Method and device for business data aggregation, electronic equipment and storage medium
CN115640310A
Assessment data processing method and device, equipment, product and readable storage medium
CN115965296A
