Data management method for leading cockpit multidimensional visualization

Abstract

The application discloses a data management method for leading cockpit multi-dimensional visualization and belongs to the technical field of computer data processing. The method comprises the following steps: collecting multi-source business data and cleaning; constructing a star multi-dimensional data model for storing and organizing hierarchical relationships in a recursive closure table; precomputing basic aggregation values and configuring dynamic calculation rules; configuring dimension linkage filtering rules, analyzing filtering conditions through a backend dimension linkage engine, expanding all descendant nodes of the selected nodes at one time by using the recursive closure table, performing aggregation, calculating derivative indexes, and comparing the derivative indexes with historical query results of the same conditions in a session-level cache to generate differential data packets containing only changed data and affected chart identifiers. The application solves the problems of long refresh time, poor data consistency and insufficient real-time monitoring capability of the existing cockpit when performing multi-dimensional filtering linkage.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing technology, specifically relating to a data management method for multi-dimensional visualization of the leadership cockpit. Background Technology

[0002] As enterprises deepen their digital transformation, the leadership dashboard, as a core component of decision support systems, needs to aggregate and display massive amounts of data from multiple business systems, such as project management, financial management, intellectual property, scientific and technological talent, and research platforms, and provide multi-dimensional analysis and real-time monitoring capabilities. The leadership dashboard not only needs to present macro-level statistical indicators but also support flexible multi-dimensional filtering, drill-down, and roll-up interactive operations to enable layer-by-layer data exploration from the group level to departments and from annual to daily levels. However, most existing dashboard implementations follow the traditional "request-response" model, revealing the following significant technical shortcomings when faced with scenarios involving high dimensions, multiple charts, and large data volumes: Multiple network round trips result in prolonged overall refresh times. A typical dashboard interface usually contains dozens of independent chart components, such as overview cards, trend charts, leaderboards, and distribution charts. When users change dimensional conditions using filters (e.g., selecting a specific region or switching time granularity), the traditional approach is for the front-end to send multiple independent data requests to the back-end. Each chart component obtains its required data slices and then renders them independently. In this model, assuming the interface contains 50 charts, at least 50 HTTP requests are generated, each going through a complete cycle of connection establishment, data transmission, and server querying. Even in a gigabit LAN environment, due to serial blocking of requests or browser concurrency limitations, the total refresh latency can easily exceed 3-5 seconds. Users will experience noticeable lag and delays after filtering, resulting in a very poor user experience.

[0003] Multiple independent requests from different sources can easily lead to data consistency issues. When each chart component requests data independently, slight differences in the time the request arrives at the backend, concurrent writes in the data processing pipeline, and different queries hitting different cache states can cause inconsistencies in the values ​​displayed for the same metric across different charts on the same page. For example, the metric "Total Number of Projects Under Development" might show 128 items in the overview card, while in the regional rankings, due to a slightly later request time and the arrival of new projects, it might show 130 items. This inconsistency severely undermines the authority and credibility of the dashboard as a basis for decision-making.

[0004] The calculation of derived metrics severely consumes computing resources. Growth rates, percentages, year-on-year comparisons, and month-on-month comparisons are core analytical elements of the dashboard. In traditional architectures, these metrics are typically calculated ad-hoc by the front-end or back-end for each request, requiring the acquisition of two sets of data—current and base period—and a full comparison. When users frequently switch filter criteria, this repetitive full calculation exhausts the OLAP engine's computing resources, leading to a sharp increase in query response time and making it difficult to guarantee end-to-end response speeds within seconds in high-concurrency scenarios. This is especially true when there are complex organizational hierarchies, requiring recursive queries of the organizational tree and layer-by-layer aggregation, further amplifying the computational overhead.

[0005] Insufficient real-time monitoring capabilities. Existing alarm mechanisms are mostly based on timed polling, where a backend scheduled task performs a statistical query every few minutes, compares it with thresholds, and then sends an alarm. This approach cannot capture rapid changes in indicators such as project progress and budget overruns, resulting in alarm delays and missed alerts. For management scenarios requiring immediate response, such as continuous project delays or sudden budget overruns, timed polling mechanisms cannot meet real-time requirements.

[0006] Therefore, how to achieve unified aggregation and standardization of multi-source heterogeneous data, and complete the synchronization and consistent refresh of all visualization charts with extremely low latency when users perform dimension filtering, while realizing streaming real-time alarms for key indicators, has become an important technical problem that urgently needs to be solved in the field of leadership cockpits. Summary of the Invention

[0007] The purpose of this invention is to provide a data management method for multi-dimensional visualization of the leadership cockpit, so as to solve the technical problems of prolonged cockpit dimension linkage refresh time, poor data consistency and insufficient real-time monitoring capabilities in the prior art.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The data management method of multi-dimensional visualization in the leadership cockpit includes the following steps: Step S1: The data acquisition engine collects business data from multiple business systems using a timed incremental approach and stores this business data in the raw layer of the data lake. The data acquisition engine supports various heterogeneous data sources, including relational databases, file systems, message queues, and application programming interfaces (APIs). The timed incremental acquisition strategy identifies changed data based on timestamps or log sequence numbers, and simultaneously utilizes a Change Data Capture (CDC) mechanism to obtain real-time change events at the database log level, ensuring data integrity and low latency. The collected raw data is partitioned and stored in the data lake, preserving its original format and providing an immutable data foundation for subsequent processing.

[0009] Step S2: Perform standardized cleaning processing on the business data, including deduplication, missing value completion, standardization of statistical calibers and units of measurement, and filtering of outliers, to obtain cleaned data. The cleaning operation uses a distributed batch processing engine to execute deterministic cleaning rules for each batch of incremental data, ensuring that the cleaned data meets predefined data quality standards and is written to the data warehouse detail layer.

[0010] Step S3: Based on the cleaned data, construct a star-shaped multidimensional data model in the data warehouse. This star-shaped multidimensional data model consists of an organizational dimension table, a time dimension table, a business dimension table, and an indicator fact table. The indicator fact table uses project identifier and time identifier as primary keys. The organizational dimension table uses a recursive closure table to store hierarchical relationships. A recursive closure table is a closure representation that describes the ancestor-descendant relationships between all pairs of nodes in a tree or network hierarchical structure. It contains three core fields: ancestor identifier, descendant identifier, and hierarchical distance. It can complete the query of all descendants of any node with constant time complexity, greatly improving the efficiency of organizational dimension aggregation and drill-down.

[0011] Step S4: Pre-calculate basic aggregate values ​​in the OLAP engine and configure dynamic calculation rules. These rules are used to calculate growth rate and percentage metrics in real time during queries. The OLAP engine employs columnar storage and vectorized calculation. The pre-calculated materialized view stores commonly used cross-dimensional aggregation results in advance. During queries, the materialized view is scanned directly or lightweight filtering is performed on it. Ratio-type metrics such as growth rate and percentage, because their denominators depend on the query context, are registered as UDF functions as dynamic calculation rules. These functions perform real-time calculations on the aggregated result set during queries, avoiding full recalculation.

[0012] Step S5: Configure dimensional linkage filtering rules. This is the core linkage mechanism of this invention, and specifically includes: The front-end interface captures the user's operation on the filter control to generate filter conditions. The filter control includes an organizational hierarchy cascading selector and a time granularity selector, and the user can combine multiple conditions. The backend dimension linkage engine parses the filtering conditions. When the filtering conditions involve organizational levels, it uses the recursive closure table to expand all descendant nodes of the selected node at once and generates an OLAP query statement containing the aggregation conditions of descendant nodes. Based on the multidimensional data model, it executes the aggregation query to obtain the filtered indicator value set. The growth rate and percentage are calculated by calling the dynamic calculation rules to obtain the original value and the derived value; The dimensional linkage engine compares the current query result with the previous query result with the same filtering conditions stored in the session-level cache, extracts the changed fields and affected chart identifiers, and generates a differential data packet. The differential data packet contains the changed index value, the set of affected chart identifiers, and the corresponding data fragments. The differential comparison is performed in depth in the form of a tree or key-value pair, and only records the paths and values ​​of the added, deleted, and modified fields, which greatly compresses the transmission volume. The differential data packet is returned to the front end, and the current query result replaces the previous query result in the session-level cache. The session-level cache uses the user session identifier and the filter condition hash value as a combined key to ensure cache isolation for different users and different filter conditions. The cache is also time-sensitive and is automatically cleared after timeout. The front-end global state manager receives the differential data packets and, based on the set of chart identifiers, sends targeted notifications to the corresponding chart components using a publish-subscribe pattern. Each chart component then performs a partial update based on the data fragments. Chart components that have not subscribed will not trigger any operation, thus achieving global synchronous refresh. This "one-request, differential response, targeted update" mechanism fundamentally solves the data inconsistency and latency problems caused by multiple requests.

[0013] Step S6: Set the data refresh cycle and anomaly alarm threshold. The refresh cycle includes two strategies: scheduled batch refresh and real-time streaming refresh based on CDC. The anomaly alarm threshold can be configured differently according to organizational level and business type, and is stored in the configuration center, supporting dynamic adjustment.

[0014] Step S7: Push the processed data to the leadership dashboard visualization interface, displaying it in preset chart formats. The visualization interface includes various chart types such as overview cards, trend charts, distribution maps, leaderboards, and alarm dashboards. Data push uses a lightweight differential data packet protocol.

[0015] Step S8: Monitor the real-time collected business data through the stream computing engine. When the metric exceeds the abnormal alarm threshold, trigger an automatic alarm and generate alarm information. The stream computing engine subscribes to the data change event stream, performs complex event processing (CEP) pattern matching, calculates aggregated metrics in real time and compares them with the threshold. Once an alarm is triggered, it is immediately pushed through the message channel and the alarm dashboard is updated simultaneously.

[0016] Preferably, the standardized cleaning process includes: performing deduplication on duplicate data, identifying duplicate records based on primary keys or business unique identifiers; filling in missing values, filling in default values, mean, mode, or markers according to business rules; unifying data statistical standards and units of measurement, for example, unifying the unit of funding to ten thousand yuan and the date format to the ISO standard; and filtering and correcting abnormal values ​​and incorrect formats, using statistical distribution or rule engines to identify outliers and correct or remove them.

[0017] The organizational dimension table includes at least the levels of group, region, subsidiary and department; the recursive closure table includes ancestor identifier, descendant identifier and hierarchical distance fields; the time dimension table includes year, quarter and month; the business dimension table includes project type, R&D direction and risk level; the metrics in the indicator fact table include quantity, amount, percentage, growth rate and score.

[0018] The growth rate indicator is calculated in real time according to the following formula: Growth rate = (Current period value - Previous period value) ÷ Previous period value × 100%; The percentage indicator is calculated in real time according to the following formula: Percentage = Individual value ÷ Total value × 100%; When the previous period value or the total value is zero, a placeholder, such as "N / A" or "-", is returned to avoid division by zero anomalies.

[0019] When a user selects an organizational level through the cascading selector, the dimension linkage engine uses the recursive closure table to query all descendant nodes of the selected node, and summarizes the indicator data of the descendant nodes to achieve aggregation including lower-level units; when the user triggers a drill-down command, it returns the aggregated data of the next level based on the recursive closure table, and records the drill-down path through breadcrumb navigation, supporting the user to roll back to the previous level.

[0020] The set data refresh cycle includes automatically extracting incremental data from the business system at fixed time intervals, obtaining database log change events through a change data capture mechanism, and updating the multidimensional data model through stream processing jobs; the abnormal alarm thresholds include project progress lag percentage threshold, overdue days threshold, and budget deviation threshold, and support differentiated configuration according to organizational level to adapt to the granularity of concern of different management levels.

[0021] The process of pushing the processed data to the leadership dashboard visualization interface and displaying it in the form of preset charts includes: displaying core indicators through overview cards, displaying changes over time through trend charts, displaying organizational or business-level composition through distribution charts, displaying regional sorting through leaderboards, and displaying a list of abnormal items through alarm dashboards; the differential data packets are compressed and transmitted using Protocol Buffers serialization format, and can also be combined with gzip for secondary compression to further reduce network bandwidth consumption.

[0022] The system monitors real-time collected business data through a stream computing engine, including: monitoring project execution progress, budget utilization rate, and report submission status; detecting continuous progress events through a complex event processing mode, calculating lag margins, and marking any indicator exceeding the corresponding threshold as a major or minor deviation based on severity, and pushing alarm notifications via a message channel. The message channel supports multiple terminals such as WeChat Work, DingTalk, and email to ensure timely delivery of alarm information.

[0023] The system collects business data from multiple business systems, and pulls data from project management, funding management, intellectual property, scientific and technological personnel, and scientific research platform systems via application programming interfaces or direct database connections through a configured timed automatic collection strategy. The OLAP engine is a distributed online analytical processing engine based on columnar storage, such as ClickHouse; the stream computing engine is a distributed stream processing engine, such as Apache Flink; and the message channel is a high-throughput distributed message queue, such as Apache Kafka.

[0024] The multidimensional data model supports data slicing, drill-down, and roll-up operations, providing analysts with flexible OLAP multidimensional analysis capabilities. The dimension-linked filtering rules allow for the combination of multiple filtering conditions, including arbitrary combinations of organizational, time, and business dimensions. Under preset test conditions, the synergy of the pre-calculated basic aggregate value, the recursive closure table query expansion, the dynamic calculation rules, the difference comparison to generate differential data packets, and the publish-subscribe partial update ensures an end-to-end response time of less than 1 second from the triggering of filtering conditions to the synchronous refresh of all charts. The preset test conditions include a fact table containing more than 100,000 project records, a visualization interface containing more than 50 chart components, a server with an OLAP engine configured with at least 8 CPU cores and 32GB of memory, and a gigabit LAN between the front-end and back-end servers. This performance indicator ensures that users receive immediate and smooth feedback for every filtering operation in complex dashboard scenarios with high data volume and high chart density, demonstrating the significant practicality of this invention.

[0025] The beneficial effects of this invention are: First, by combining OLAP pre-aggregation with query-time dynamic calculation, the calculation of derived indicators is decomposed into pre-calculation of basic aggregation and query-time UDF calculation of ratio indicators. This significantly reduces the recalculation overhead of derived indicators when filtering dimensions, ensuring query performance in high-concurrency scenarios. As a result, the system can still maintain a stable response latency even when more than 50 chart components are refreshed at the same time.

[0026] Second, a recursive closure table is used to achieve infinite-level drill-down and sub-level summaries in the organizational dimension, solving the path splicing problem of data aggregation queries under complex organizational structures. Traditional recursive CTE queries experience a sharp performance drop when the hierarchy is deep, while the closure table trades space for time, requiring only a single equality query to obtain all descendant nodes, reducing the complexity of organizational dimension aggregation queries from O(d*n) to O(1), where d is the recursion depth.

[0027] Third, through a coordinated mechanism of "unified backend query, session-level cache difference comparison to generate differential data packets, and frontend publish-subscribe targeted partial updates," multiple network requests are merged into a single request, transmitting only the changed data, significantly reducing network traffic and frontend rendering overhead. The differential comparison and caching mechanism ensures consistency in the synchronous refresh of multiple charts, eliminating discrepancies in the same metric values ​​caused by multiple independent requests. In typical scenarios, network transmission volume is reduced by an average of 60%, frontend rendering time is reduced by 70%, and end-to-end response time is consistently less than 1 second.

[0028] Fourth, by leveraging CDC and streaming computing, real-time monitoring of metrics and CEP alerts are achieved, transforming passive polling into proactive push notifications and significantly shortening the risk response window. The streaming computing engine can capture data changes at the second level and accurately identify abnormal patterns such as continuous progress delays and sudden increases in funding by defining complex pattern matching rules, and generate tiered alerts in real time, giving managers valuable decision-making time. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall process of the multi-dimensional visualization data management method for the leadership cockpit of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0031] Example 1; This embodiment provides a multi-dimensional visualization data management method for a leadership dashboard, applied to a science and technology innovation project management dashboard. It aims to achieve unified data aggregation across systems, flexible multi-dimensional filtering and analysis, and real-time project risk monitoring. The overall process is as follows: Figure 1 As shown, it includes steps S1 to S8.

[0032] Step S1: Unified collection of multi-source science and technology innovation data.

[0033] The data acquisition engine collects data from six business systems—project management, funding management, intellectual property, scientific and technological talent, research platform, and project monitoring—using a timed incremental approach. The acquisition task is configured to trigger a full initialization and subsequent incremental acquisition at 2:00 AM daily. Incremental acquisition retrieves records changed since the last acquisition based on the timestamp field (e.g., update_time) of each source table. To capture near real-time data changes, a Debezium connector is deployed to monitor the binlog logs of each source database, pushing the insertion, update, and deletion events of each row of data to a Kafka message queue in real time. The collected raw data is written verbatim to the data lake's raw layer in JSON or Avro format, and stored in partitions according to business system and date to ensure data traceability and replayability. This step successfully solved the problem of unified access to multi-source heterogeneous data, avoiding data silos between different business systems.

[0034] Step S2: Data standardization and cleaning process.

[0035] The incremental data collected daily at midnight is subjected to standardized cleaning using an Apache Spark batch job. The cleaning rules are encapsulated as a configurable rule set, specifically including: Deduplication: For duplicate project records in the project management system that may be generated due to interface retries, deduplication is performed based on a combination of project number and update time, retaining the latest version.

[0036] Missing value completion: For the missing "Patentee" field of some patents in the intellectual property table, it is completed from the project master data table based on the associated project number; for the missing "Budget Amount" in the funding data, if its historical months have values, it is filled with the most recent valid value.

[0037] Unify statistical standards and units of measurement: unify the unit of financial indicators to "ten thousand yuan", unify the date format to "yyyy-MM-dd", and unify the project type code to be mapped to the standard dictionary.

[0038] Anomaly filtering: Use box plots to identify obvious erroneous records with funding arrival rates greater than 100% or negative, mark them as anomalies and remove them, and record anomaly logs for manual review.

[0039] The cleaned data is stored in the data warehouse detail layer in ORC columnar format to ensure efficient subsequent analysis and querying.

[0040] Step S3: Construct a star-shaped multidimensional data model.

[0041] Create a star schema in the data warehouse. The core fact table is `fact_project_kpi`, with primary keys `project_id` and `snapshot_date`. It includes metrics such as: number of projects initiated, number of projects under development, total budget, funds received, total expenditure, and number of days overdue. Three dimension tables are built around the fact table: The organization dimension table `dim_org` stores hierarchical node information such as group, region, subsidiary, and department. To support efficient hierarchical traversal, an additional recursive closure table `org_closure` is created, with fields including `ancestor_id` (ancestor node ID), `descendant_id` (descendant node ID), and `distance` (hierarchical distance). For example, if a subsidiary belongs to the East China region, and the East China region belongs to the group headquarters, then the closure table will insert records such as (Group Headquarters ID, Subsidiary ID, 2) and (East China Region ID, Subsidiary ID, 1), including self-references (distance=0).

[0042] The time dimension table dim_time contains attributes such as date_key, year, quarter, month, and week.

[0043] The business dimension table dim_biz includes classification attributes such as project type, R&D direction, and risk level.

[0044] This star schema has a clear structure, is easy to extend, and forms the basis for subsequent OLAP queries.

[0045] Step S4: Configure OLAP pre-computation and dynamic calculation rules.

[0046] Cleaned data from the data warehouse is loaded into the ClickHouse OLAP engine using a batch import tool. To accelerate frequently used queries, a materialized view, `mv_base_agg`, is created to pre-calculate basic aggregate values ​​such as the number of projects initiated, the number of projects under development, and the total budget, grouped by region, year, and business type. ClickHouse's materialized views can automatically update incrementally as underlying data is inserted, ensuring the freshness of the pre-aggregated data. Simultaneously, user-defined functions (UDFs) `calc_growth` and `calc_ratio` are registered in ClickHouse to calculate the growth rate and percentage, respectively. These UDFs receive aggregated data blocks as input and perform vectorized calculations, avoiding the need to pull data to the application layer for further calculations, significantly reducing data transfer and transformation overhead.

[0047] Step S5: Configure the dimension-linked filtering rules, which is the core innovation of this embodiment.

[0048] The front-end uses the Vue framework, and the global state manager uses Vuex. The visual interface contains more than 50 chart components, each of which registers a subscription with Vuex upon mounting to listen to its associated state fragments. The filter control area provides organizational hierarchy cascading selectors and time granularity selectors. After the user interacts with the filter, the front-end collects all currently valid filter conditions (e.g., Region = "East China", Time Granularity = "Quarter"), encapsulates them into a JSON object, and sends a single HTTP POST request via Axios to the back-end dimension linkage engine service.

[0049] The dimensional linkage engine is an independent microservice built on Spring Boot. After receiving the filtering conditions, it executes the following process: (1) Condition Parsing and Query Generation: If the filtering conditions include an organizational dimension, the engine accesses the org_closure closure table and executes the pre-compiled SQL: "SELECT descendant_id FROM org_closure WHERE ancestor_id= ?". This query returns a list of IDs for the selected node and all its descendant nodes at once. For example, selecting "East China Region" not only retrieves the ID of the East China Region itself, but also the IDs of all its subsidiaries and departments. The engine uses these IDs as parameters in the IN clause and embeds them into the WHERE condition of the fact table query to achieve data aggregation including subordinate units. If no organizational condition is selected, this filter is not added.

[0050] (2) OLAP query execution: Construct a complete ClickHouse SQL query to obtain filtered basic metric values ​​from materialized views or the underlying fact table. The query fully leverages ClickHouse's columnar storage, vectorized execution, and materialized view acceleration to ensure that aggregate queries are completed within 100 milliseconds.

[0051] (3) Calculation of dynamic derived indicators: After obtaining the basic aggregation result set, the calc_growth and calc_ratio UDFs are called inside ClickHouse to calculate derived indicators such as growth rate and percentage, and obtain the current complete set of original and derived values, R_current. For example, for each project type, the number of projects under research in the current period and the number of projects under research in the previous period are calculated to obtain the growth rate of the number of projects under research; the percentage of its funding to the total funding is calculated.

[0052] (4) Comparison of Session-Level Cache Differences: The Dimensional Linkage Engine maintains a Redis cluster as a session-level cache. The cache key is formed by concatenating the user session ID and the MD5 hash value of the filter condition JSON string, and the value is the serialized byte string of the query result with the same filter condition from the previous query. The engine calculates the hash of the current filter condition and checks whether the corresponding R_last exists in the cache.

[0053] If R_last exists, perform a deep difference comparison. The comparison algorithm recursively traverses both indicator trees, recording all changed field paths (e.g., "East China Region.Number of Projects Under Research.value", "East China Region.Funding Growth Rate.displayValue") and their new values. If any indicator value that a chart component depends on changes, that chart ID is added to the set of affected charts. The difference comparison generates a difference data packet, DiffPacket, with the following structure: json { "charts": ["card_total_project", "chart_trend_revenue", ...], “deltas”: { “card_total_project”: { “in_project_count”: 128, “yoy_growth”: “12.5%”}, “chart_trend_revenue”: { “Q2”: 4560.5, “Q3”: 4890.2} } } If R_last does not exist (either during the first query or the cache has expired), then DiffPacket contains all query results, and the charts list contains all chart labels.

[0054] (5) Overwrite cache and return: The engine serializes R_current and stores it in the Redis cache, overwriting the value of the corresponding key. The DiffPacket is serialized using the Protocol Buffers protocol and compressed with gzip before being returned to the front end.

[0055] After receiving the DiffPacket, the Vuex global state manager iterates through the charts list and publishes update messages to each chart component via the event bus. These messages carry the corresponding deltas data fragments. Upon receiving the data fragments, the subscription callbacks of each chart component trigger a component-level partial re-render using Vue's reactive data assignment mechanism (e.g., `this.chartData = newData`). Chart components not appearing in the charts list are completely unaffected and do not perform any rendering operations.

[0056] This linkage mechanism merges 50+ requests in the traditional model into 1 request, reducing network transmission volume by an average of 60% due to the transmission of only differential components. The front end only updates the changed components, reducing rendering overhead by 70%. Actual end-to-end latency (from user clicking filter to all charts refreshing) is consistently below 800 milliseconds, far surpassing existing technologies.

[0057] Step S6: Set the data refresh cycle and the abnormal alarm threshold.

[0058] In the system configuration center, the data service layer cache expiration time is set to 1 hour, supporting manual forced refresh from the front end. Alarm threshold configurations include: a project progress delay percentage >20% is considered a major deviation, 10%~20% is a minor deviation; overdue days ≥30 days are considered a major alarm, 7~29 days are a minor alarm; budget deviation >15% is considered a major deviation. These thresholds can be configured differently by region, as different regions have different business rhythms and budget flexibility.

[0059] Step S7: Visualization.

[0060] The cockpit homepage displays key indicators such as "Total Number of Projects Under Development," "Total Annual Budget," "Fund Execution Rate," and "Total Personnel Input" through overview cards. Card data is derived from global status and refreshes dynamically with each filter. A trend chart component displays project additions and completion trends over the past 12 months. The leaderboard displays the top 5 regions by the number of projects initiated.

[0061] Step S8: Real-time monitoring and automatic alarm.

[0062] The Flink job consumes the real-time change event stream captured by Debezium in step S1 of Kafka. For example, project progress reporting events include fields such as `project_id`, `progress_percentage`, and `report_time`. Flink CEP defines a pattern: detecting when the `progress_percentage` growth for the same `project_id` is lower than the expected threshold (e.g., less than 5% growth within a month) in two consecutive progress events. Once a match is found, the lag is calculated, compared to the alarm threshold, and an alarm record is generated and written to the MySQL alarm table. Alarm notifications are also pushed via WeChat Work robot and email service. Simultaneously, the alarm record is pushed to the front-end WebSocket long connection, refreshing the alarm dashboard in real time. Flink's state backend guarantees exactly-once semantics, ensuring that alarms are not lost or duplicated.

[0063] This embodiment achieves unified data aggregation, multi-dimensional model construction, high-performance dimension linkage filtering and synchronous refresh, and full-link data management of streaming real-time alarms through the synergy of the above steps, which is a significant improvement.

[0064] Example 2; This embodiment, based on Embodiment 1, further refines the drill-down interaction at the organizational dimension and optimizes the query of the recursive closure table. A new `level` field is added to the `org_closure` table to represent the hierarchy of the ancestor node itself (e.g., Group=0, Region=1) to support some front-end display requirements. Simultaneously, to support the "include subordinate" and "drill-down" operations of the cascading selector, two standard query interfaces are provided: When a user selects "East China Region" and checks "Include Subordinates", the following query is executed: SELECT descendant_id FROM org_closure WHERE ancestor_id = ?. This query returns all descendants, achieving upward data aggregation (i.e., summarizing from subsidiaries to the region).

[0065] When a user double-clicks at the regional level to trigger drill-down, the front-end sends a drill-down command carrying the current node ID. Upon receiving the drill-down command, the dimensional linkage engine executes the query: `SELECT descendant_id FROM org_closure WHEREancestor_id = ? AND distance = 1`. This query only returns direct child nodes (subsidiaries). The engine further queries the pre-aggregated base values ​​of these child nodes to form the next level of aggregated data. Simultaneously, the engine includes a breadcrumb navigation path in the response, such as "Group > East China Region > Shanghai Subsidiary," which the front-end updates the breadcrumb component accordingly. Because drill-down only involves queries with `distance=1` and the data volume is relatively small, the response latency remains consistently below 100 milliseconds. Furthermore, the drill-down operation also follows a session-level cache difference comparison mechanism, only sending down the changed parts, making interface refresh extremely agile.

[0066] Example 3; This embodiment focuses on further ensuring end-to-end response time and providing an in-depth description of the technical means to verify the achievement of the performance indicators in claim 10. The hardware environment is as follows: the OLAP engine ClickHouse is deployed on a single Intel Xeon 8-core CPU @2.5GHz, 32GB DDR4 memory, and 1TB NVMe SSD server; the front-end Nginx and the back-end API gateway are connected via Gigabit Ethernet, with an average network latency of less than 0.5ms. Test data: 150,000 project records were injected into the fact table, covering 5 dimensions. The front-end loads 55 chart components, including 20 cards, 10 trend charts, 10 bar charts, 10 tables, and 5 map components.

[0067] The optimization measures adopted include: First, the front-end requests use WebSocket long connections to avoid the overhead of repeated handshakes in HTTP short connections, and support proactive push from the server.

[0068] Second, the Vuex global state manager internally uses the immer immutable data structure library. When differential data packets arrive and update the state tree, immer can accurately detect which state fragments have changed, and only require components that depend on these fragments to re-render, avoiding the performance bottleneck of conventional deep comparisons.

[0069] Third, ClickHouse creates projection and materialized skip indexes for organization dimension filtering and closure table queries, ensuring that queries can still be completed within 100 milliseconds even if the IN clause contains hundreds of descendant node IDs.

[0070] Fourth, the differential comparison generation algorithm uses field-level hash comparison. The hash of each indicator value is stored in the cache in the previous query result R_last. During comparison, only the hash of the corresponding indicator in R_current needs to be calculated and compared one by one, with a time complexity of O(n). Once a difference in hash is found, the change is recorded. This algorithm can still complete the process extremely quickly when the number of indicators reaches tens of thousands.

[0071] Through the above methods, multiple rounds of automated scripts simulated frequent switching of filtering conditions, and the 95th percentile of the end-to-end response time was 780 milliseconds, with a maximum value of 920 milliseconds, which fully meets the design requirement of less than 1 second.

[0072] Example 4; This embodiment expands the depth of calculation and data cleaning for business dimension indicators. In the intellectual property dimension, the raw data comes from the intellectual property management system. In the data standardization and cleaning step S2, in addition to routine cleaning, quality indicators such as the number of patent citations and the number of patent families are obtained by calling an external patent database API. These indicators are calculated using the weighted formula "Patent Comprehensive Score = Base Score + Number of Citations × 2 + Number of Patent Families × 1.5" during the data warehouse ETL process and stored as a metric field in the fact table. In the dynamic calculation rules of the OLAP engine, a separate UDF function for the proportion of high-value patents (score > 80) is configured. The growth rate calculation is also applied to indicators such as the year-on-year growth of patent grants. In the talent dimension, the educational background mapping is unified during cleaning, unifying "Master's Degree" and "Master" as "Master," and calculating composite indicators such as the proportion of those with master's degrees or above and the proportion of those with senior professional titles. These calculation rules are all registered via UDFs, requiring no additional pre-aggregation, and offering high flexibility.

[0073] Example 5; This embodiment represents a lightweight modification to the mobile management dashboard, ensuring smooth operation on mobile devices. In the data push phase, the data service layer generates lightweight differential packets for mobile devices, removing visualization-irrelevant fields (such as highlight color configurations for certain charts and animation parameters specific to large screens). The differential data packets are compressed using gzip on top of ProtoBuf serialization, resulting in an average data volume reduction of 70% due to the smaller screen size and fewer charts on mobile devices. The front-end state manager implements a progressive update strategy: prioritizing the refresh of the top visible cards and alert bars on the first screen, then using the Intersection Observer API to monitor the viewport and delay loading and updating chart components outside the screen. Simultaneously, network policies are adaptive: maintaining full data updates and 1-minute polling in WiFi environments, and automatically reducing the refresh frequency to 5 minutes in cellular networks, pushing only critical data such as alerts. These modifications ensure that the dashboard's first screen loading time is controlled within 2 seconds and the filter refresh latency is less than 1.5 seconds under 4G network conditions, guaranteeing usability in mobile management scenarios.

[0074] Example 6; This embodiment describes the implementation scheme in a SaaS multi-tenant environment. All data entities (data lake raw layer tables, data warehouse dimension tables, fact tables, and ClickHouse tables) are assigned a tenant_id partition key. In the data acquisition engine, each tenant's acquisition tasks are configured in isolation. The OLAP cube is physically sharded and stored with tenant_id as the first dimension, ensuring physical data isolation between tenants. When a tenant's dimension-linked filtering request passes through the API gateway, the tenant_id is automatically parsed from the JWT token and injected into the backend service context. When processing filtering conditions, the dimension-linked engine automatically adds "tenant_id = ?" to the WHERE clause of all OLAP queries, routing the query to the corresponding shard. Alarm thresholds support tenant-level customization, with each tenant's JSON configuration maintained by the configuration center. Flink jobs dynamically load the application upon startup and configuration changes. This solution has supported the stable operation of over 200 tenant instances, with complete data isolation between tenants and no performance impact, verifying the scalability and multi-tenant adaptability of this invention.

[0075] In summary, the multi-dimensional visualization data management method for the leadership cockpit provided by this invention systematically solves the pain points of existing technologies, such as slow cockpit linkage refresh, data inconsistency, and untimely monitoring, through a series of means including data acquisition, standardization, star schema and recursive closure table design, hybrid acceleration of pre-computation and dynamic computation, a "one-request-difference comparison-targeted update" mechanism for dimension linkage filtering, and streaming real-time alarms. Its technical solution is novel in conception, rigorous in steps, and possesses outstanding substantive features and significant progress, demonstrating a high degree of creativity. It has been verified as feasible and exhibits excellent performance in various business scenarios and environments, possessing extremely high practicality and novelty.

[0076] The method provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multi-dimensional visualization data management method for a leadership cockpit, characterized in that, Includes the following steps: The data acquisition engine collects business data from multiple business systems in a timed incremental manner and stores the business data in the raw layer of the data lake. The business data is subjected to standardized cleaning processing, including deduplication, missing value completion, unification of statistical caliber and measurement unit, and filtering of abnormal values, to obtain cleaned data; Based on the cleaned data, a star-shaped multidimensional data model is constructed in the data warehouse. The star-shaped multidimensional data model consists of an organizational dimension table, a time dimension table, a business dimension table, and an indicator fact table. The indicator fact table uses the project identifier and time identifier as primary keys, and the organizational dimension table uses a recursive closure table to store the hierarchical relationship. The basic aggregate values ​​are pre-calculated in the OLAP engine, and dynamic calculation rules are configured to calculate the growth rate and percentage indicators in real time during the query. Configure dimensional linkage filtering rules, which include: the front-end interface captures user operations on the filtering control to generate filtering conditions, the filtering control including an organizational hierarchy cascading selector and a time granularity selector; the back-end dimensional linkage engine parses the filtering conditions, and when the filtering conditions involve organizational hierarchy, it uses the recursive closure table to expand all descendant nodes of the selected node at once, and generates an OLAP query statement containing descendant node aggregation conditions, and executes the aggregation query based on the multidimensional data model to obtain the filtered indicator value set; it calls the dynamic calculation rules to calculate the growth rate and percentage, and obtains the original value and derived value; the dimensional linkage engine then... The query result is compared with the previous query result stored in the session-level cache with the same filtering conditions. The changed fields and affected chart identifiers are extracted to generate a differential data package. The differential data package contains the changed index value, the set of affected chart identifiers, and the corresponding data fragments. The differential data package is returned to the front end, and the current query result replaces the previous query result in the session-level cache. The front end global state manager receives the differential data package and, based on the set of chart identifiers, sends targeted notifications to the corresponding chart components using a publish-subscribe pattern. Each chart component performs a local update based on the data fragments, achieving global synchronous refresh. Set the data refresh cycle and anomaly alarm threshold; The processed data is pushed to the leader's dashboard visualization interface and displayed in the form of preset charts; The system monitors real-time collected business data through a stream computing engine. When the metrics exceed the abnormal alarm threshold, it triggers an automatic alarm and generates alarm information.

2. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The standardized cleaning process includes: removing duplicate data, filling in missing values, standardizing data statistical standards and units of measurement, and filtering and correcting abnormal values ​​and incorrect formats.

3. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The organizational dimension table includes at least the levels of group, region, subsidiary and department; the recursive closure table includes ancestor identifier, descendant identifier and hierarchical distance fields; the time dimension table includes year, quarter and month; the business dimension table includes project type, R&D direction and risk level; the metrics in the indicator fact table include quantity, amount, percentage, growth rate and score.

4. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The growth rate indicator is calculated in real time according to the following formula: Growth rate = (Current period value - Previous period value) ÷ Previous period value × 100%; The percentage indicator is calculated in real time according to the following formula: Percentage = Individual value ÷ Total value × 100%; When the previous period value or the total value is zero, a placeholder is returned.

5. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, When a user selects an organizational level through the cascading selector, the dimension linkage engine uses the recursive closure table to query all descendant nodes of the selected node, and summarizes the indicator data of the descendant nodes to achieve aggregation including lower-level units; when the user triggers a drill-down command, it returns the aggregated data of the next level based on the recursive closure table, and records the drill-down path through breadcrumb navigation.

6. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The set data refresh cycle includes automatically extracting incremental data from the business system at fixed time intervals, obtaining database log change events through a change data capture mechanism, and updating the multidimensional data model through stream processing jobs; the abnormal alarm thresholds include project progress lag percentage threshold, overdue days threshold, and budget deviation threshold, and support differentiated configuration according to organizational level.

7. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The process of pushing the processed data to the leadership dashboard visualization interface and displaying it in the form of preset charts includes: displaying core indicators through overview cards, displaying changes over time through trend charts, displaying organizational or business-dimensional composition through distribution charts, displaying regional sorting through leaderboards, and displaying a list of abnormal projects through alarm dashboards; the differential data packets are compressed and transmitted using Protocol Buffers serialization format.

8. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The monitoring of real-time collected business data through the stream computing engine includes: monitoring project execution progress, fund utilization rate, and report submission status; detecting continuous progress events through complex event processing mode, calculating the lag range, and marking any indicator as a major deviation or a minor deviation according to its severity when any indicator exceeds the corresponding threshold, and pushing alarm notifications via message channel.

9. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The system collects business data from multiple business systems and pulls data from project management, funding management, intellectual property, scientific and technological personnel, and scientific research platform systems via application programming interfaces or direct database connections by configuring a timed automatic collection strategy. The OLAP engine is a distributed online analytical processing engine based on columnar storage, the stream computing engine is a distributed stream processing engine, and the message channel is a high-throughput distributed message queue.

10. The data management method for multi-dimensional visualization of the leadership cockpit according to claim 1, characterized in that, The multidimensional data model supports data slicing, drill-down, and roll-up operations. The dimension-linked filtering rules allow for the combination of multiple filtering conditions. Under preset test conditions, the synergy of the pre-calculated basic aggregate value, the recursive closure table query expansion, the dynamic calculation rules, the difference comparison to generate differential data packets, and the publish-subscribe mode local update ensures that the end-to-end response time from the triggering of filtering conditions to the completion of the synchronous refresh of all charts is less than 1 second. The preset test conditions are: the fact table contains more than 100,000 project records, the visualization interface contains more than 50 chart components, the server configuration for deploying the OLAP engine is not less than 8 core CPUs and 32GB of memory, and the front-end server and the back-end server are connected by a gigabit LAN.

Images