Data view hierarchical aggregation modeling method and system
Through the data view hierarchical aggregation modeling method, through three stages of data modeling, hierarchical multi-dimensional aggregation model and data view structure, the core challenges of big data processing and analysis are solved, and efficient and accurate data processing and interactive methods are achieved. The generation of data views improves data utilization and decision-making support capabilities.
Patent Information
- Application Number
- CN202411174693.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2025-02-07
AI Technical Summary
In the era of big data, how to effectively process, analyze and form professional business models and present these data has become a core challenge in the field of data science.
A data view hierarchical aggregation modeling method is proposed, and the hierarchical aggregation of data and flexible configuration of views are realized through three stages: data modeling, hierarchical multi-dimensional aggregation model and data view structure. The specific steps include data preprocessing using business model power technology, hierarchical view generation using GBDT algorithm, extracting the aggregation model and combining business modules, and finally generating an interactive data view through vue-draggable-resizable-porkys technology.
It reduces the complexity of data processing, improves the efficiency and accuracy of data processing, enhances the scalability and flexibility of the system, supports users to generate data views with interactive functions, and improves data utilization and decision-making support capabilities.
Smart Images
Figure CN119396920A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer data technology, and in particular to a data view hierarchical aggregation modeling method and system. Background Art
[0002] With the advent of the big data era, data in various usage scenarios has exploded and data types are becoming increasingly complex. How to effectively process, analyze, form professional business models and present these data has become a core challenge in the field of data science. The data view hierarchical aggregation modeling solution uses the key technology of hierarchical aggregation model to reduce the complexity of data, quickly generate high-precision data models, and display and analyze them through flexibly configured data views, thereby improving data utilization and decision support capabilities to adapt to future changes and developments in the system. Summary of the invention
[0003] The present invention proposes a data view hierarchical aggregation modeling method, comprising:
[0004] S1. Data modeling: For the data entities and their attributes that need to be processed, the data modeling process is completed through four stages: data preprocessing, data source association, feature processing, and feature transformation. In the data preprocessing process, the business model power technology solution is mainly used, that is, the power of the business model features is used to enhance the model's ability to fit the data;
[0005] S2. Hierarchical multi-dimensional aggregation model. Hierarchical multi-dimensional aggregation modeling as the core part mainly includes hierarchical view generation, extraction aggregation model, and business module combination;
[0006] S3, data view structure, based on the data interface provided by the aggregation model and combined with the configuration of the view layout module, supports users to generate data views with interactive functions.
[0007] Furthermore, in the data preprocessing process, the business model power technology solution is mainly used, and the specific calculation formula is:
[0008] y=θ 0 +θ 1 x+θ 2 x 2 +θ 3 x 3 +…+θ n x n Among them, y is the predicted value, x is the independent variable, and θ 0 ,θ 1 ,θ 2 ,θ 3 ,θ n are the parameters of the model; the independent variables are the original features or the transformed features.
[0009] Furthermore, in S2, the hierarchical view is generated by using a GBDT comprehensive hierarchical algorithm to separate the data into native attributes, business attributes, performance-aware attributes, and security-aware attributes for attribute stratification.
[0010] Furthermore, the GBDT algorithm formula is as follows:
[0011] where f 0 (x), is the initial model, usually set to 0, J m is the number of leaf nodes of the decision tree at the mth iteration, I(x∈R jm ) is an indicator function that takes the value 1 when x belongs to the jth leaf node in the mth iteration and 0 otherwise.
[0012] Furthermore, in S2, the extracted aggregation model is divided into view generation technology, according to the data layered architecture, through group aggregation, compression aggregation, approximate aggregation, distributed aggregation, and real-time aggregation, to form modules or data elements that can work independently. By extracting the aggregation model, an interface of an independent unit that can be developed, tested and maintained independently is formed. The system can be flexibly adjusted according to different needs and respond quickly. The solution can be generated.
[0013] Furthermore, in S2, the business module combination is a business module combination technical solution that mainly receives the generated aggregation model, and filters, sorts and combines views according to business needs to form a business module set with specific logic and business meanings, supports encapsulation and reuse of business modules, and improves the scalability and maintainability of the system.
[0014] Furthermore, in the S3, the data view structure supports users to interact with the data view by clicking, dragging, zooming and other operations through the vue-draggable-resizable-porkys technology to generate the data view, and can support in-depth development of the data view in multiple languages through codemirror.
[0015] Furthermore, the flexible construction configuration of the data view enables users to conduct in-depth mining and visual display of the data view.
[0016] In another aspect, a data view hierarchical aggregation modeling system includes:
[0017] Data modeling module: For the data entities and their attributes that need to be processed, the data modeling process is completed through four stages: data preprocessing, data source association, feature processing, and feature transformation. In the data preprocessing process, the business model power technology solution is mainly used, that is, the power of the business model features is used to enhance the model's ability to fit the data;
[0018] Hierarchical multi-dimensional aggregation model module: Hierarchical multi-dimensional aggregation modeling as the core part mainly includes hierarchical view generation, extraction of aggregation model, and business module combination;
[0019] Data view structure module: Based on the data interface provided by the aggregation model and combined with the configuration of the view layout module, it supports users to generate data views by interacting with the data views.
[0020] On the other hand, the present invention provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed by a computer, the computer performs the steps of any of the above-mentioned data view hierarchical aggregation modeling methods.
[0021] The beneficial effects of the embodiments of the present invention are as follows:
[0022] The present invention uses an innovative hierarchical aggregation design concept to decompose complex data processing tasks into multiple dimensions. This processing method reduces the complexity of data processing, improves the efficiency and accuracy of data processing, standardizes the data flow and interaction between various levels, and comprehensively improves the scalability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A schematic diagram of a data view hierarchical aggregation modeling method flow chart provided by an embodiment of the present invention;
[0024] Figure 2 A schematic diagram of a data view hierarchical aggregation modeling system provided by an embodiment of the present invention;
[0025] Figure 3 A schematic diagram of the composition of a computer storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] Certain embodiments of the present invention will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments are shown. Indeed, various embodiments of the present invention may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure may satisfy applicable legal requirements.
[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
[0028] See also Figure 1 The present invention provides a data view hierarchical aggregation modeling method, comprising:
[0029] S1. Data modeling: For the data entities and their attributes that need to be processed, the data modeling process is completed through four stages: data preprocessing, data source association, feature processing, and feature transformation. In the data preprocessing process, the business model power technology solution is mainly used, that is, the power of the business model features is used to enhance the model's ability to fit the data;
[0030] S2. Hierarchical multi-dimensional aggregation model. Hierarchical multi-dimensional aggregation modeling as the core part mainly includes hierarchical view generation, extraction aggregation model, and business module combination;
[0031] S3, data view structure, based on the data interface provided by the aggregation model and combined with the configuration of the view layout module, supports users to generate data views with interactive functions.
[0032] Data views are divided into two types: interactive and non-interactive. This technical solution is superior to generating data views with interactive effects.
[0033] Specifically, in the data preprocessing process, the business model power technology solution is mainly used for implementation, and the specific calculation formula is:
[0034] y=θ 0 +θ 1 x+θ 2 x 2 +θ 3 x 3 +…+θ n x n Among them, y is the predicted value, x is the independent variable, and θ 0 ,θ 1 ,θ 2 ,θ 3 ,θ n are the parameters of the model; the independent variables are the original features or the transformed features.
[0035] Specifically, in S2, the hierarchical view is generated by using the GBDT comprehensive hierarchical algorithm to separate the data into native attributes, business attributes, performance-aware attributes, and security-aware attributes for attribute stratification.
[0036] Specifically, the GBDT algorithm formula is as follows:
[0037] Among them, f 0 (x) is the initial model, usually set to 0, J m is the number of leaf nodes of the decision tree at the mth iteration, I(x∈R jm ) is an indicator function that takes the value 1 when x belongs to the jth leaf node in the mth iteration and 0 otherwise.
[0038] Specifically, in S2, the extracted aggregation model is divided into view generation technology. According to the data layered architecture, it is aggregated into modules or data elements that can work independently through grouping aggregation, compression aggregation, approximate aggregation, distributed aggregation, and real-time aggregation. By extracting the aggregation model, an interface of an independent unit that can be developed, tested and maintained independently is formed. The system can be flexibly adjusted according to different needs and a response plan can be quickly generated.
[0039] System Integration Apifox defines clear interfaces for each module or data element, including input, output, dependencies, etc., forming an independent unit that can be developed, tested and maintained independently
[0040] Specifically, in S2, the business module combination is a business module combination technical solution that mainly receives the generated aggregation model, and filters, sorts and combines views according to business needs to form a set of business modules with specific logic and business meanings, supports the encapsulation and reuse of business modules, and improves the scalability and maintainability of the system.
[0041] Specifically, in S3, the data view structure supports users to interact with the data view by clicking, dragging, zooming and other operations through the vue-draggable-resizable-porkys technology to generate the data view, and can support in-depth development of the data view in multiple languages through codemirror.
[0042] Specifically, the flexible construction configuration of the data view enables users to conduct in-depth mining and visual display of the data view.
[0043] On the other hand, see Figure 2 The present invention also provides a data view hierarchical aggregation modeling system, including:
[0044] Data modeling module: For the data entities and their attributes that need to be processed, the data modeling process is completed through four stages: data preprocessing, data source association, feature processing, and feature transformation. In the data preprocessing process, the business model power technology solution is mainly used, that is, the power of the business model features is used to enhance the model's ability to fit the data;
[0045] Hierarchical multi-dimensional aggregation model module: Hierarchical multi-dimensional aggregation modeling as the core part mainly includes hierarchical view generation, extraction of aggregation model, and business module combination;
[0046] Data view structure module: Based on the data interface provided by the aggregation model and combined with the configuration of the view layout module, it supports users to generate data views by interacting with the data views.
[0047] On the other hand, see Figure 3 The present invention provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the steps of the aforementioned data view hierarchical aggregation modeling method.
[0048] In addition, the above-mentioned figures are only schematic illustrations of the processing included in the method according to the embodiment of the present invention, and are not intended to limit the purpose. The order in the figures does not limit the time sequence of these operations, and these operations can be performed simultaneously or independently in multiple modules. Moreover, multiple steps can be combined into one step for execution, or one step can be disassembled into multiple steps for execution.
[0049] The above description is only a specific implementation mode of the present invention and is not intended to limit the protection scope of the present invention. For those skilled in the art, the above embodiments can be changed or modified within the scope of the present invention, and these changes and modifications also fall within the protection scope of the claims of the present invention.
Claims
1. A data view hierarchical aggregation modeling method, characterized in that: include: S1. Data modeling: For the data entities and their attributes that need to be processed, the data modeling process is completed through four stages: data preprocessing, data source association, feature processing, and feature transformation. In the data preprocessing process, the business model power technology solution is mainly used, that is, the power of the business model features is used to enhance the model's ability to fit the data; S2. Hierarchical multi-dimensional aggregation model. Hierarchical multi-dimensional aggregation modeling as the core part mainly includes hierarchical view generation, extraction aggregation model, and business module combination; S3, data view structure, based on the data interface provided by the aggregation model and combined with the configuration of the view layout module, supports users to generate data views with interactive functions.
2. The data view hierarchical aggregation modeling method according to claim 1, characterized in that: In S1, the data preprocessing process is mainly implemented by using the business model power technology solution, and the specific calculation formula is: y=θ0+θ1x+θ2x 2 +θ3x 3 +…·+θ n x n Among them, y is the predicted value, x is the independent variable, θ0, θ1, θ2, θ3, θ n are the parameters of the model; the independent variables are the original features or the transformed features.
3. The data view hierarchical aggregation modeling method according to claim 1, characterized in that: In S2, the hierarchical view is generated by using the GBDT comprehensive hierarchical algorithm to separate the data into native attributes, business attributes, performance-aware attributes, and security-aware attributes for attribute stratification.
4. The data view hierarchical aggregation modeling method according to claim 3 is characterized in that: The GBDT algorithm formula is as follows: Where f0(x) is the initial model, usually set to 0, J m is the number of leaf nodes of the decision tree at the mth iteration, I(x∈R jm ) is an indicator function that takes the value 1 when x belongs to the jth leaf node in the mth iteration and 0 otherwise.
5. The data view hierarchical aggregation modeling method according to claim 4 is characterized in that: In S2, the extracted aggregation model is divided into view generation technology. According to the data layered architecture, it is aggregated into modules or data elements that can work independently through grouping aggregation, compression aggregation, approximate aggregation, distributed aggregation, and real-time aggregation. By extracting the aggregation model, an interface of an independent unit that can be developed, tested and maintained independently is formed. The system can be flexibly adjusted according to different needs and a response plan can be generated quickly.
6. The data view hierarchical aggregation modeling method according to claim 5, characterized in that: In S2, the business module combination is a business module combination technical solution that mainly receives the generated aggregation model, and filters, sorts and combines views according to business needs to form a set of business modules with specific logic and business meanings, supports encapsulation and reuse of business modules, and improves the scalability and maintainability of the system.
7. The data view hierarchical aggregation modeling method according to claim 1, characterized in that: In the S3, the data view structure supports users to The vue-draggable-resizable-porkys technology implements click, drag, zoom and other operations to interact with the data view, generates the data view, and supports in-depth development of the data view in multiple languages through codemirror.
8. The data view hierarchical aggregation modeling method according to claim 7, characterized in that: The flexible construction configuration of the data view enables users to conduct in-depth mining and visual display of the data view.
9. A data view hierarchical aggregation modeling system, characterized in that: include: Data modeling module: For the data entities and their attributes that need to be processed, the data modeling process is completed through four stages: data preprocessing, data source association, feature processing, and feature transformation. In the data preprocessing process, the business model power technology solution is mainly used, that is, the power of the business model features is used to enhance the model's ability to fit the data; Hierarchical multi-dimensional aggregation model module: Hierarchical multi-dimensional aggregation modeling as the core part mainly includes hierarchical view generation, extraction of aggregation model, and business module combination; Data view structure module: Based on the data interface provided by the aggregation model and combined with the configuration of the view layout module, it supports users to generate data views with interactive functions.
10. A computer-readable storage medium storing computer program instructions, wherein when the computer program instructions are executed by a computer, the computer executes the steps of the data view hierarchical aggregation modeling method according to any one of claims 1 to 8.