Data Acquisition and Visualization Processing System Based on Big Data Analysis

Abstract

This invention discloses a data acquisition and visualization processing system based on big data analysis, belonging to the field of data processing technology. It includes a resource allocation component that collects system load data and business demand data in real time, and weights and fuses system load, business demand, and resource elasticity coefficients to generate dynamic resource quantities. A visualization quality assessment component monitors resource utilization efficiency and data loading efficiency in real time, and dynamically adjusts the visualization rendering quality score by combining the error degree between the quantified visualization results and the original data. A dynamic threshold adjustment component analyzes visualization quality and system resource status in real time, adjusts the data hierarchy division threshold through a closed-loop feedback mechanism, and generates an adaptive threshold adjustment strategy based on differences in visualization quality and resource elasticity. This invention continuously optimizes resource allocation and data processing strategies based on visualization quality and resource status by improving resource utilization, enhancing visualization quality, strengthening system stability, and improving self-optimization capabilities.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically a data acquisition and visualization processing system based on big data analysis. Background Technology

[0002] In the field of digital data processing technology, data acquisition and visualization systems based on big data analytics are one of the core technologies for realizing data-driven decision-making. Furthermore, with the rapid development of big data technology, data acquisition and visualization systems have been widely applied in fields such as finance, e-commerce, healthcare, and transportation.

[0003] Currently, data acquisition and visualization processing systems based on big data analytics mainly employ traditional resource allocation and visualization rendering techniques. While these techniques achieve data acquisition and visualization to a certain extent, they have many shortcomings in terms of system resource utilization, visualization quality, and system stability. For example, traditional resource allocation techniques typically use fixed resource allocation strategies, making it difficult to dynamically adjust resource allocation based on system load and business needs; and traditional visualization rendering techniques typically use fixed rendering parameters, making it difficult to dynamically adjust visualization quality based on system resource status and business requirements. Summary of the Invention

[0004] The purpose of this invention is to provide a data acquisition and visualization processing system based on big data analysis, which solves the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution, including: The resource allocation component is configured to collect system load data and business demand data in real time, and to weight and fuse system load, business demand and resource elasticity coefficient to generate dynamic resource quantity; The parallelism adjustment component is configured to obtain the dynamic resource quantity of the current system, including CPU idle rate, memory idle rate, and remaining IO bandwidth. Based on the preset resource quantity-parallelism mapping table, query the target parallelism corresponding to the dynamic resource quantity; Furthermore, when the amount of dynamic resources is higher than the first preset threshold, the parallelism of data processing is increased to the target parallelism; when the amount of dynamic resources is lower than the second preset threshold, the parallelism of data processing is decreased to the target parallelism, wherein the first preset threshold is higher than the second preset threshold. The visualization quality assessment component is configured to monitor resource utilization efficiency and data loading efficiency in real time, and dynamically adjust the visualization rendering quality score by combining the degree of error between the quantitative visualization results and the original data. The dynamic threshold adjustment component is configured to analyze visualization quality and system resource status in real time, adjust the data hierarchy partitioning threshold through a closed-loop feedback mechanism to obtain the updated data hierarchy partitioning threshold, and generate an adaptive threshold adjustment strategy based on the differences in visualization quality and resource elasticity. The visualization quality assessment component further includes: The system includes a resource efficiency monitoring module, a loading efficiency monitoring module, and a quality assessment calculation module, and the formula for calculating the visualization rendering quality score is as follows: ; In the formula: VRRQ is the visualization rendering quality score, VFT is the visual fidelity threshold, R is the dynamic resource quantity, RL is the system load resource quantity, S is the actual data step size, S0 is the maximum loading step size, and w1 and w2 are both weight coefficients.

[0006] Optionally, the resource allocation component includes: The load awareness module is configured to collect system load data, including CPU, memory, and disk I / O, in real time, and calculate the system load resource amount based on rendering priority using a sliding window algorithm. The demand response module is configured to collect business request quantity and data processing volume business demand data in real time, and calculate the business demand resource quantity based on queue length using a weighted average method. The elastic computing module is configured to calculate the resource elasticity coefficient based on the maximum resource elasticity coefficient, the minimum resource elasticity coefficient, and the CPU utilization rate. The resource quantity calculation module is configured to calculate the dynamic resource quantity based on the system load resource quantity, the demand resource quantity, and the resource elasticity coefficient.

[0007] Optionally, the specific method by which the resource allocation component calculates the dynamic resource quantity is as follows: Multiply the system load resource quantity by the resource allocation weight coefficient to obtain the first weighted impact value of system load on resource allocation; Based on the required resource quantity and resource allocation weight coefficient, the second weighted impact value of business demand on resource allocation is calculated. The first weighted impact value and the second weighted impact value are added together to obtain the total weighted resource; the total weighted resource is multiplied by the resource elasticity coefficient to obtain the dynamic resource quantity that can be allocated to the current system. The resource allocation weight coefficient is determined based on the analytic hierarchy process (AHP) pre-set for the main application scenarios of the system.

[0008] Optionally, the visualization quality assessment component includes: The resource efficiency monitoring module is configured to receive the dynamic resource quantity and the system load resource quantity calculated by the resource allocation component, and calculate the resource utilization efficiency by the ratio method; The loading efficiency monitoring module is configured to collect data loading speed in real time and calculate data loading efficiency using a ratio method. The quality assessment calculation module is configured to weight and fuse resource utilization efficiency and data loading efficiency to obtain a comprehensive visualization efficiency value, and then multiply the comprehensive visualization efficiency value by a visual fidelity threshold to obtain the visualization rendering quality score. The visual fidelity threshold was determined through user surveys and experimental testing, with different threshold ranges corresponding to different types of visualization tasks.

[0009] Optionally, the dynamic threshold adjustment component includes: The quality difference analysis module is configured to analyze the degree of deviation of the visualization quality based on the visualization rendering quality score and the target rendering quality score; The resource difference analysis module is configured to analyze the degree of deviation of the system's resource elasticity based on the resource elasticity coefficient and the optimal resource elasticity coefficient; The threshold adjustment calculation module is configured to calculate the adjusted updated data level division threshold based on the degree of deviation between visualization quality and system resource elasticity, as well as the current data level division threshold.

[0010] Optionally, the specific method for the dynamic threshold adjustment component to calculate the updated data hierarchy partitioning threshold adjustment is as follows: Multiply the first difference between the resource elasticity coefficient and the optimal resource elasticity coefficient by the first adjustment coefficient, and add the benchmark value to obtain the quality correction factor. Multiply the second difference between the resource elasticity coefficient and the optimal resource elasticity coefficient by the second adjustment coefficient, and then subtract the second adjustment coefficient from the benchmark value to obtain the resource correction factor; The updated data level division threshold is obtained by multiplying the data level division threshold by the quality correction factor and then by the resource correction factor.

[0011] Optionally, the system further includes a visualization rendering component, configured to perform visualization rendering using an adaptive rendering algorithm based on the visualization rendering quality score calculated by the visualization quality assessment component. The visualization rendering component includes: The adaptive rendering module is configured to dynamically adjust rendering parameters, including resolution, frame rate, and color depth, based on the visual rendering quality score. The visualization type switching module is configured to switch visualization types according to user needs and data types. It adopts a template-based design and supports the rapid generation of different types of visualization results.

[0012] Optionally, the resource allocation component, the visualization quality assessment component, and the dynamic threshold adjustment component achieve collaborative optimization through a closed-loop feedback mechanism: The dynamic threshold adjustment component adjusts the data hierarchy partitioning threshold based on visualization quality and system resource status. Changes in the data hierarchy partitioning threshold will affect the calculation of the business requirement data and the dynamic resource quantity. The resource allocation component adjusts the resource allocation scheme according to the dynamic resource quantity. The change in the resource allocation scheme affects the resource utilization efficiency, the data loading efficiency, and the calculation of the visualization rendering quality score. The visualization quality assessment component adjusts the visualization rendering parameters based on the visualization rendering quality score. Changes in the visualization rendering parameters affect the visualization quality, system resource status, and the calculation of the updated data hierarchy division threshold.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention enables the system to dynamically adjust resource allocation based on system load and business needs through resource allocation calculations by a resource allocation component. When the system load is low, the system can appropriately reduce resource allocation to avoid resource waste; when the system load is high, the system can appropriately increase resource allocation to ensure stable system operation. The calculated resource elasticity coefficient can be dynamically adjusted according to system load and business needs, achieving both rationality and flexibility in resource allocation.

[0014] This invention utilizes a visualization quality assessment component to calculate visualization quality, enabling the system to dynamically adjust visualization quality based on system resource status and business needs. When system resources are sufficient, the system can appropriately increase visualization quality, providing clearer and more accurate visualization results; when system resources are insufficient, the system can appropriately decrease visualization quality to ensure stable system operation. The calculated visual fidelity threshold can be dynamically adjusted according to system resource status and business needs, achieving both rationality and flexibility in visualization quality to improve the visualization quality after data analysis and processing.

[0015] Third, this invention adjusts the data hierarchy partitioning threshold through a dynamic threshold adjustment component, enabling the system to dynamically adjust the data hierarchy partitioning threshold based on visualization quality and system resource status. When visualization quality is high, the system can appropriately lower the data hierarchy partitioning threshold to reduce data processing volume and improve system stability; when visualization quality is low, the system can appropriately raise the data hierarchy partitioning threshold to increase data processing volume and improve visualization quality. Specifically, the calculated quality correction factor and resource correction factor can be dynamically adjusted according to visualization quality and system resource status, achieving both rationality and flexibility in the data hierarchy partitioning threshold.

[0016] Fourth, this invention enables the system to achieve self-optimization capabilities through the synergistic effect of the resource allocation component, the visual quality assessment component, and the dynamic threshold adjustment component. When business requirements change, the system can dynamically adjust system parameters based on the resource allocation calculations of the resource allocation component, the visual quality calculations of the visual quality assessment component, and the threshold adjustments based on the data hierarchy of the dynamic threshold adjustment component, thereby achieving system self-optimization. Attached Figure Description

[0017] Figure 1 This is a flowchart of the data acquisition and visualization processing process of the big data analysis-based data acquisition and visualization processing system of the present invention. Figure 2 This is a flowchart of the data processing procedure for visualization rendering in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1, such as Figure 1 and Figure 2 As shown, this implementation provides a data acquisition and visualization processing system based on big data analytics, including: The resource allocation component is configured to collect system load data and business demand data in real time, and to weight and fuse system load, business demand and resource elasticity coefficient to generate dynamic resource quantity; The parallelism adjustment component is configured to adjust the parallelism of data processing based on dynamic resource availability. The visualization quality assessment component is configured to monitor resource utilization efficiency and data loading efficiency in real time, and dynamically adjust the visualization rendering quality score by combining the degree of error between the quantitative visualization results and the original data. The dynamic threshold adjustment component is configured to analyze visualization quality and system resource status in real time, adjust the data hierarchy partitioning threshold through a closed-loop feedback mechanism to obtain the updated data hierarchy partitioning threshold, and generate an adaptive threshold adjustment strategy based on differences in visualization quality and resource elasticity.

[0020] In the above embodiments, the resource allocation component collects system load indicators such as CPU, memory, and disk I / O in real time through the load awareness module, and combines this with business demand data to generate dynamic resource quantities through a weighted fusion algorithm, achieving dynamic and adaptive resource allocation. The parallelism adjustment component allocates computing nodes and cores to different data analysis tasks based on the dynamic resource quantities, dynamically adjusting the parallelism of data processing and optimizing big data processing efficiency. The visualization quality assessment component monitors resource utilization efficiency and data loading efficiency in real time, and dynamically calculates the visualization rendering quality score based on the visual fidelity threshold, providing a quantitative basis for visualization rendering. The dynamic threshold adjustment component analyzes the current visualization rendering quality score and the system resource elasticity status, and uses a closed-loop feedback mechanism to dynamically adjust the data hierarchy partitioning threshold, achieving an adaptive balance between data storage, processing, and visualization quality.

[0021] Through the collaborative work of the aforementioned components, this embodiment achieves dynamic resource allocation and adaptive quality control throughout the entire data acquisition, processing, and visualization process. When system load or business requirements change, the system can automatically adjust resource allocation strategies, data processing parallelism, visualization rendering quality, and data hierarchy thresholds. This optimizes resource utilization, improves data analysis efficiency and visualization quality, and enhances the system's self-optimization capabilities and robustness in the face of complex and ever-changing business scenarios, all while ensuring stable system operation.

[0022] Example 2, as Figure 1 As shown, based on Embodiment 1, the resource allocation component provided in this embodiment of the invention includes: The load awareness module is configured to collect system load data, including CPU, memory, and disk I / O, in real time, and calculate the system load resource amount based on rendering priority using a sliding window algorithm. The demand response module is configured to collect business demand data such as the number of business requests and the amount of data processed in real time, and calculate the amount of resources required by the business based on the queue length using a weighted average method. The elastic computing module is configured to calculate the resource elasticity coefficient based on the maximum resource elasticity coefficient, the minimum resource elasticity coefficient, and the CPU utilization rate. The resource quantity calculation module is configured to calculate dynamic resource quantity based on system load resource quantity, demand resource quantity, and resource elasticity coefficient.

[0023] The core of the load awareness module, as we understand it, lies in transforming raw system load metrics into a numerical value reflecting the amount of resources the system can currently use to support high-priority data rendering. Using a sliding window algorithm (e.g., calculating the average of the last N sampling periods), it smooths out instantaneous fluctuations, obtaining a stable and reliable system load resource quantity. The demand response module focuses on the pending business itself, quantifying the total computing resource demand of the current business by monitoring metrics such as task queue length and the data volume of a single task. The elastic computing module introduces the concept of a resource elasticity coefficient, whose value dynamically changes with the system's CPU utilization, reflecting the flexibility of resource allocation under the current load. When CPU utilization is low, the elasticity coefficient is high, allowing the system to flexibly allocate redundant resources to improve performance; when CPU utilization is high, the elasticity coefficient is low, and the system tends to allocate resources conservatively to ensure stability.

[0024] In this embodiment, the formula for calculating dynamic resource quantity is as follows: ; In the formula: R represents the actual amount of dynamic resources allocated; REC is the resource elasticity coefficient, which is specifically obtained through... Calculations show that in this formula, REC max REC is the maximum resource elasticity coefficient. min The minimum resource elasticity coefficient (REC) is used to measure the flexibility of resource allocation under the current load. (CPU is the CPU utilization rate, ensuring a smooth transition in resource elasticity.) max and REC min These are preset elastic boundary values ​​for the system design phase, corresponding to the states when CPU utilization is 0% and 100%, respectively. For example, when the system is idle, CPU utilization is 0%, and the system is in its maximum elastic state, REC=0.8, indicating that the system can flexibly allocate 80% of redundant resources to improve performance. When the system is fully loaded, CPU utilization is 100%, and the system is in its minimum elastic state, prioritizing stability, REC=0.3, indicating that the system retains only 30% resource elasticity to ensure stability. Intermediate states are smoothly transitioned through linear interpolation, ensuring the continuity and stability of resource allocation.

[0025] RL represents the system load resource amount based on rendering priority, and RL = rendering priority coefficient × total system resources. The rendering priority coefficient can be predefined according to the business importance of the data, for example, set in the system as follows: Core transaction data: Priority coefficient = 0.9; User behavior data: Priority coefficient = 0.6; System log data: Priority coefficient = 0.3; Total system resources = Number of CPU cores × Processing capacity of a single core (number of operations per second) × Time (seconds); RQ is the required resource quantity based on the queue length, and RQ = Data level division threshold × Quantity of data to be processed × Resource requirement for unit data processing. The data level division threshold divides levels according to the analysis value of the data. For example, it can be set in the system at the initial stage: Core data layer: Threshold = 0.9 (only includes the most critical 10% of the data); Key data layer: Threshold = 0.6 (includes the first 40% of important data); Ordinary data layer: Threshold = 0.3 (includes all data); k is the resource allocation weight coefficient, which can be preset according to the main application scenarios of the system. For example: Real-time monitoring system: k = 0.8 (prioritize ensuring the rendering priority); Offline analysis system: k = 0.3 (prioritize ensuring resource utilization rate); Hybrid scenario system: k = 0.5 (balance real-time performance and utilization rate).

[0026] RL × k: Represents the resource allocation based on business importance. The larger the k value, the more the processing of high-priority data is prioritized; RQ × (1 - k): Represents the resource allocation based on the actual data volume. The larger the (1 - k) value, the more the throughput of data processing is prioritized; (RL × k + RQ × (1 - k)): Reflects the optimal resource allocation benchmark value of the system under the current business requirements and load status. Among them, RL × k and RQ × (1 - k) use the same resource allocation weight coefficient k, which can ensure the unified balance logic of business priority and system load. And by adjusting a single parameter - the resource allocation weight coefficient k, the resource allocation strategy of the system can be quickly changed. In other words, k + (1 - k) = 1, which can also ensure the total control of resource allocation and avoid over-allocation or under-allocation of resources: When k = 1, the result = RL, indicating that resources are allocated completely according to business priority, which is applicable to scenarios with extremely high real-time requirements; When k = 0, the result = RQ, indicating that resources are allocated completely according to the data volume, which is applicable to scenarios with extremely high throughput requirements; When 0 < k < 1, the result is the weighted average of the two, which is applicable to most balanced scenarios.

[0027] Thus, through the collaborative calculations of the four sub-modules mentioned above, the resource allocation component can generate a dynamic resource quantity value R that comprehensively considers the system's real-time load, business pressure, resource elasticity, and business strategies. This value serves as the core basis for the entire system's resource allocation, directly driving subsequent parallelism adjustments and visualized quality control, thereby achieving closed-loop resource management from perception to decision-making.

[0028] Example 3, as Figure 1 As shown, based on Embodiment 2, the parallelism adjustment component provided in this embodiment of the invention is configured to obtain the dynamic resource quantity of the current system, including CPU idle rate, memory idle rate, and remaining IO bandwidth. Based on the preset resource quantity-parallelism mapping table, query the target parallelism corresponding to the dynamic resource quantity; Furthermore, when the amount of dynamic resources is higher than the first preset threshold, the parallelism of data processing is increased to the target parallelism; when the amount of dynamic resources is lower than the second preset threshold, the parallelism of data processing is decreased to the target parallelism, and the first preset threshold is higher than the second preset threshold.

[0029] Understandably, in big data processing, parallelism is a key parameter affecting task execution efficiency. Excessive parallelism can lead to increased resource contention and scheduling overhead, while insufficient parallelism fails to fully utilize system resources. The goal of this component is to allocate appropriate computing resources to each data analysis task based on the currently available dynamic resource quantity R, and to determine its optimal parallelism.

[0030] In this embodiment, the specific adjustment method of the parallelism adjustment component is as follows: The system maintains a task pool, where each task has a preset unit task resource requirement (UTRN). When a task needs to be scheduled, the component allocates computing nodes and cores to the task based on the current dynamic resource quantity R. The allocation strategy follows the principle of "more allocation when resources are sufficient, less allocation when resources are scarce." Simultaneously, the component dynamically decides whether to enable computationally intensive algorithms (such as deep learning models) or switch to lightweight algorithms (such as statistical analysis) based on the allocated resource quantity, to match resource supply. Specifically, computing nodes are allocated to different analysis tasks based on the dynamic resource quantity R; tasks with higher dynamic resource quantity R receive more computing cores. When the dynamic resource quantity R is sufficient, complex analysis algorithms (such as deep learning models) are enabled; when the dynamic resource quantity R is insufficient, lightweight algorithms (such as statistical analysis) are switched to.

[0031] Furthermore, the parallelism adjustment directly affects the processing efficiency of big data analysis and must be completed before the visualization quality assessment component begins analysis.

[0032] Therefore, the formula for calculating the degree of parallelism is: PAR=min(PARmax ,int(R / UTRN)); In the formula: PAR represents parallelism. max R represents the maximum parallelism, R represents the actual dynamic resource allocation, and UTRN represents the resource requirement per unit task.

[0033] The resource quantity-parallelism mapping table was obtained in advance through testing: Under different CPU / memory / IO resource configurations, the data processing throughput and resource utilization of the system were tested when the parallelism was n (n is a positive integer). The parallelism that satisfies the condition of having a resource utilization of less than 80% and the highest throughput was selected as the target parallelism for the corresponding resource range.

[0034] For example, when the CPU idle rate is ≥70% and the memory idle rate is ≥60%, the corresponding parallelism is 16; when the CPU idle rate is between 40% and 70% and the memory idle rate is between 30% and 60%, the corresponding parallelism is 8; when the CPU idle rate is <40% or the memory idle rate is <30%, the corresponding parallelism is 2.

[0035] The system collects dynamic resource data every 5 seconds. If the resource data collected in three consecutive data collections all meet the threshold requirements of the same parallelism range, then the parallelism is adjusted to avoid system fluctuations caused by frequent adjustments.

[0036] At this point, the parallelism adjustment component has completed fine-grained resource management during the data processing phase. It adjusts before the visualization process begins, ensuring optimal resource allocation for the data analysis phase and laying a solid foundation for generating high-quality visualization results. This dynamic parallelism adjustment mechanism based on real-time resource provisioning significantly improves the overall throughput and response speed of big data processing tasks.

[0037] Example 4, as Figure 1 As shown, based on Example 3, the visualization quality assessment component provided in this embodiment of the invention includes: The resource efficiency monitoring module is configured to receive the dynamic resource quantity and system load resource quantity calculated by the resource allocation component, and calculate the resource utilization efficiency by the ratio method. The loading efficiency monitoring module is configured to collect data loading speed in real time and calculate data loading efficiency using a ratio method. The quality assessment calculation module is configured to weight and fuse resource utilization efficiency and data loading efficiency to obtain a comprehensive visualization efficiency value, and then multiply the comprehensive visualization efficiency value by the visual fidelity threshold to obtain a visualization rendering quality score.

[0038] Understandably, the quality of visualization rendering depends not only on the rendering algorithm itself but also on the efficiency of underlying resource and data supply. The resource efficiency monitoring module assesses the proportion of resources actually used for visualization relative to the system's baseline resources available for visualization by calculating resource utilization efficiency; a higher proportion indicates more abundant resources. The loading efficiency monitoring module assesses the timeliness of data supply by calculating data loading efficiency; a higher ratio indicates smoother data loading. These two efficiency metrics reflect the health of the visualization rendering support environment from the dimensions of "computing power supply" and "data supply," respectively.

[0039] In this embodiment, the quality assessment calculation module weights and fuses the two efficiency indicators mentioned above to form a comprehensive efficiency assessment value. Then, this comprehensive value is multiplied by a predefined visual fidelity threshold to obtain the final visualization rendering quality score. The visual fidelity threshold represents the maximum error range between the user-acceptable visualization result and the original data for a specific type of data (such as core transaction data, user behavior data, etc.). Its calculation formula is as follows: ; In the formula: VRRQ is a rating for visual rendering quality. VFT is the visual fidelity threshold, which can be predefined based on the business importance of the data, for example: Core transaction data: 0.02, ensuring pixel-level accuracy; Key business data: 0.05, ensuring details are preserved; Standard statistics: 0.1, to ensure trend display; R / RL represents the ratio of actual resource usage to baseline resource demand, reflecting the adequacy of resource utilization. S is the actual data step size that is actually loaded, S0 is the maximum loading step size, and S / S0 reflects the throughput of data processing. w1 and w2 are both weight coefficients, and w1+w2=1. For example, w1 is set to 0.7 and w2 to 0.3. Among them, resource utilization efficiency R / RL has a greater impact on visualization quality, so it is assigned a higher weight. Data loading efficiency S / S0 has a relatively smaller impact on visualization quality, so it is assigned a lower weight. The specific weight settings can be flexibly set according to the degree of impact in the actual processing. Multiply by VFT It can assess the degree of error in visualization results, providing a basis for adjusting the threshold for data hierarchy division.

[0040] Thus, the Visual Representation Rendering Quality (VRRQ) score, derived by integrating resource utilization efficiency and data loading efficiency, reflects the theoretical quality of the visualization output under the current system state. The Visual Fidelity Threshold (VFT) is preset based on data type; for example, it is set to 0.02 for core transaction data to ensure high-fidelity reproduction. The weighting coefficients w1 and w2 can be dynamically adjusted according to business scenarios; for example, w1 can be set to 0.7 in a real-time monitoring system to emphasize the impact of resource utilization on quality. The visualization quality assessment component enables dynamic and quantitative evaluation of visualization rendering quality. This VRRQ score not only reflects the theoretical upper limit of the current rendering output quality (defined by VFT) but also incorporates the real-time system performance status (reflected by a comprehensive efficiency value), providing precise and actionable quality input signals for downstream visualization rendering components and dynamic threshold adjustment components.

[0041] Example 5, as Figure 1 and Figure 2 As shown, based on Embodiment 4, the visualization quality assessment component provided in this embodiment of the invention further includes a visualization rendering component. The visualization rendering component is configured to perform visualization rendering using an adaptive rendering algorithm based on the visualization rendering quality score calculated by the visualization quality assessment component. Understandably, fixed rendering parameters cannot adapt to dynamically changing system resources and quality requirements. The core of the visualization rendering component lies in dynamically switching rendering strategies based on VRRQ scores to achieve the optimal balance between "quality," "performance," and "resource consumption."

[0042] In this embodiment, the visualization rendering component includes two sub-modules: Adaptive rendering module: Dynamically adjusts rendering parameters based on the VRRQ score range. For example: (a) Visualization rendering quality score VRRQ < 0.03 (high fidelity requirement) Applicable scenarios: core transaction data, financial statements, key business indicators Visualization processing strategy: Enables a high-quality rendering engine, supporting advanced features such as anti-aliasing, lighting effects, and gradient fill; displays precise values, detailed labels, and complete time series, supporting data drill-down and multi-dimensional analysis; supports fine-grained operations such as displaying detailed information on mouse hover, zooming in by selecting boxes, and exporting data; updates in real time (5-10 times per second) to ensure data timeliness; prioritizes CPU / GPU resources, allocating at least 80% of available rendering resources.

[0043] (ii) 0.03 ≤ Visual Rendering Quality Rating (VRRQ) < 0.07 (Medium fidelity requirement) Applicable scenarios: User behavior analysis, market trend forecasting, business operation monitoring Visualization strategy: Enable a balanced rendering engine, support basic visual effects, and disable unnecessary advanced features; display key values, key labels, and trend curves, and support data filtering and simple drill-down; support common operations such as displaying key information on mouse hover, zooming, and data export; near real-time updates (2-5 times per second), balancing timeliness and resource consumption; allocate 50-80% of available rendering resources to balance quality and performance.

[0044] (iii) Visualization rendering quality score VRRQ≥0.07 (low fidelity requirement) Applicable scenarios: System log monitoring, traffic statistics, macro trend display Visualization strategy: Enable the fast rendering engine, disable all unnecessary visual effects, and prioritize rendering speed; display statistical results, trend lines, and simplified labels, showing only core information; support basic zoom and filter operations to simplify the interaction process; update regularly (every 10-30 seconds) to reduce resource consumption; allocate ≤50% of available rendering resources to prioritize system stability.

[0045] The visualization type switching module allows users to quickly switch between visualization chart types (such as line charts, bar charts, heatmaps, etc.) from a pre-set template library based on user interaction commands or data types. This module employs a template-based design to ensure that visualizations of the correct type and quality can be generated quickly at different rendering quality levels.

[0046] At this point, the visualization rendering component has completed a closed loop from "quality score" to "actual rendering effect." It enables the system to intelligently adjust the detail and smoothness of the final visualization content presented to the user based on real-time resource conditions and quality assessment results, achieving adaptive optimization between user experience and system load.

[0047] Example 6, as Figure 1 As shown, based on Embodiment 5, the dynamic threshold adjustment component provided in this embodiment of the invention includes: The quality difference analysis module is configured to analyze the degree of deviation in visual quality based on the visual rendering quality score and the target rendering quality score. The resource difference analysis module is configured to analyze the degree of deviation of the system's resource elasticity based on the resource elasticity coefficient and the optimal resource elasticity coefficient; The threshold adjustment calculation module is configured to calculate the adjusted updated data level division threshold based on the degree of deviation between visualization quality and system resource elasticity, as well as the current data level division threshold.

[0048] Understandably, the data hierarchy threshold determines the proportion of data classified as high-priority "core data" for fine-grained processing and high-quality rendering. This threshold directly impacts data processing volume, storage costs, analysis speed, and final visualization quality. The goal of the dynamic threshold adjustment component is to dynamically adjust the data hierarchy threshold based on the gap between the current system performance (visualization quality, resource elasticity) and the ideal state (target quality, optimal elasticity), allowing the system to automatically approach its optimal operating point.

[0049] In this embodiment, the updated data hierarchy partitioning threshold is calculated using a formula that incorporates dual feedback: ; In the formula: DHL new The updated data hierarchy partitioning threshold, calculated as described above, can dynamically adjust the data hierarchy partitioning threshold DHL based on the current visualization quality and system resource status, thereby achieving system self-optimization. DHL is the data hierarchy partitioning threshold currently configured and used for hierarchical partitioning in the system. VRRQ0 is the target rendering quality score. The calculation involves improving visualization quality by expanding the scope of core data and increasing processing resources for high-priority data when the visualization quality fails to meet the target. The correction factor is set to 0.2 to avoid system instability caused by over-adjustment, so that every 0.01 quality difference corresponds to a 0.2% threshold adjustment, achieving fine-tuning; REC opt The optimal resource elasticity coefficient. The calculation involves reducing resource consumption by narrowing the scope of core data and decreasing the amount of high-priority data when system resources are strained. The correction factor is set to 0.1 to ensure that the system remains stable when resources fluctuate, so that every 0.1 elasticity difference corresponds to a 1% threshold adjustment, achieving robust adjustment.

[0050] Furthermore, the updated data hierarchy segmentation threshold DHL new It also includes a dynamic adjustment mechanism for data storage tiers, for example: When the updated data hierarchy segmentation threshold DHL new When the value is less than 0.3: the core data range is less than 30%, and only the most critical data is stored in the core storage area; When 0.3 ≤ the updated data hierarchy partitioning threshold DHL new When the value is less than 0.7: 30%-70% of the core data should be stored in the critical storage area; When the updated data hierarchy segmentation threshold DHLnew When the value is ≥0.7: the core data range is ≥70%, and most of the data is stored in the core storage area.

[0051] So, regarding the coordination of data acquisition and storage: When the updated data hierarchy segmentation threshold DHL new When the value increases, the scope of core data is expanded, and the scope of core storage area is also expanded to ensure fast access to core data; When the updated data hierarchy segmentation threshold DHL new When the value decreases, the core data range is reduced, and the core storage area is also reduced to lower storage costs. Regarding the collaboration between data analysis and storage: When the updated data hierarchy segmentation threshold DHL new When the value increases, the analysis priority of core data is increased, while ensuring that core data is stored in high-speed storage media to improve analysis efficiency; When the updated data hierarchy segmentation threshold DHL new When the value decreases, the analysis priority of core data is reduced, while non-core data is stored in low-cost storage media to reduce analysis costs.

[0052] Thus, the dynamic threshold adjustment component has implemented the system's core self-optimization mechanism. It establishes a closed-loop feedback: visualization quality and resource status affect threshold adjustment, and threshold adjustment, in turn, affects resource requirements (business requirement data) and visualization quality by changing the data hierarchy. This dynamic adjustment ensures that the system can appropriately lower quality requirements to maintain stability when resources are scarce, and improve quality to optimize the experience when resources are abundant, achieving adaptive balance and continuous optimization in long-term operation.

[0053] Example 7, based on Examples 1-6, further illustrates the collaborative optimization process achieved through a closed-loop feedback mechanism among the resource allocation component, the visualization quality assessment component, the dynamic threshold adjustment component, and the visualization rendering component.

[0054] Understandably, the core advantage of this system lies not in the functionality of individual components, but in the collaborative closed loop formed between multiple components, which possesses self-regulating capabilities. This closed loop ensures that the system can perform global and interconnected optimization when faced with fluctuations in internal resources and changes in external demands.

[0055] In this embodiment, the collaborative optimization process is specifically manifested as follows: Trigger: Changes in system load or the arrival of new business requests trigger the resource allocation component to recalculate the dynamic resource quantity R.

[0056] Resource reallocation: The parallelism adjustment component adjusts the data processing parallelism based on the new dynamic resource amount R. The resource reallocation result also affects the resource utilization efficiency R / RL in the visualization quality assessment component.

[0057] Quality Reassessment: The visual quality assessment component combines the latest resource utilization efficiency and real-time data loading efficiency to calculate a new visual rendering quality score (VRRQ).

[0058] Adaptive rendering: The visualization rendering component dynamically adjusts rendering parameters and resource usage based on the new Visual Rendering Quality Rating (VRRQ) to generate visualization results of corresponding quality levels.

[0059] Threshold readjustment: The dynamic threshold readjustment component captures the current Visual Render Quality Rating (VRRQ) and Resource Elasticity Coefficient (REC), compares them with the target value, and calculates the updated data hierarchy partitioning threshold (DHL). new .

[0060] Feedback loop: Updated data hierarchy segmentation threshold DHL new This will directly affect the classification (core / non-core), storage strategy, and calculation of business demand resource quantity (RQ) in the demand response module of subsequent data collection, thus forming a complete feedback loop and starting a new round of system status perception and adjustment.

[0061] Thus, the entire system constitutes a dynamically balanced organic whole. Any change in the state of any component will trigger adaptive adjustments in other components through pre-defined linkages, ultimately ensuring that the system operates near Pareto optimality across multiple objectives, including resource utilization, processing efficiency, visualization quality, and system stability. This design significantly enhances the intelligence and robustness of big data analytics-based data acquisition and visualization processing systems in complex and dynamic environments.

[0062] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data acquisition, visualization, and processing system based on big data analytics, characterized in that: include: The resource allocation component is configured to collect system load data and business demand data in real time, and to weight and fuse system load, business demand and resource elasticity coefficient to generate dynamic resource quantity; The parallelism adjustment component is configured to obtain the dynamic resource quantity of the current system, including CPU idle rate, memory idle rate, and remaining IO bandwidth. Based on the preset resource quantity-parallelism mapping table, query the target parallelism corresponding to the dynamic resource quantity; Furthermore, when the amount of dynamic resources exceeds the first preset threshold, the parallelism of data processing is increased to the target parallelism. When the amount of dynamic resources is lower than the second preset threshold, the parallelism of data processing is reduced to the target parallelism, where the first preset threshold is higher than the second preset threshold. The visualization quality assessment component is configured to monitor resource utilization efficiency and data loading efficiency in real time, and dynamically adjust the visualization rendering quality score by combining the degree of error between the quantitative visualization results and the original data. The dynamic threshold adjustment component is configured to analyze visualization quality and system resource status in real time, adjust the data hierarchy partitioning threshold through a closed-loop feedback mechanism to obtain the updated data hierarchy partitioning threshold, and generate an adaptive threshold adjustment strategy based on the differences in visualization quality and resource elasticity. The visualization quality assessment component further includes: The system includes a resource efficiency monitoring module, a loading efficiency monitoring module, and a quality assessment calculation module, and the formula for calculating the visualization rendering quality score is as follows: ； In the formula: VRRQ is the visualization rendering quality score, VFT is the visual fidelity threshold, R is the dynamic resource quantity, RL is the system load resource quantity, S is the actual data step size, S0 is the maximum loading step size, and w1 and w2 are both weight coefficients.

2. The data acquisition and visualization processing system based on big data analysis according to claim 1, characterized in that, The formula for calculating the dynamic resource quantity in the resource allocation component is as follows: ； In the formula: R represents the actual amount of dynamic resources allocated; REC is the resource elasticity coefficient, which is obtained through... Calculations show that REC max REC is the maximum resource elasticity coefficient. min The minimum resource elasticity coefficients are preset based on system hardware configuration and application scenarios, with values ​​ranging from 0.8 to 0.3, respectively, corresponding to the resource elasticity limits under system idle and full load conditions. CPU utilization is defined as follows: when CPU utilization is 0%, REC=0.8, and the system is in a state of maximum elasticity; when CPU utilization is 100%, REC=0.3, and the system is in a state of minimum elasticity; intermediate states are achieved through linear interpolation to smoothly transition resource elasticity. RL represents the system load resource amount based on rendering priority, RQ represents the required resource amount based on queue length, and k is the resource allocation weight coefficient.

3. The data acquisition and visualization processing system based on big data analysis according to claim 2, characterized in that: The visualization quality assessment component includes: The resource efficiency monitoring module is configured to receive the dynamic resource quantity and the system load resource quantity calculated by the resource allocation component, and calculate the resource utilization efficiency by the ratio method; The loading efficiency monitoring module is configured to collect data loading speed in real time and calculate data loading efficiency using a ratio method. The quality assessment calculation module is configured to weight and fuse resource utilization efficiency and data loading efficiency to obtain a comprehensive visualization efficiency value, and then multiply the comprehensive visualization efficiency value by a visual fidelity threshold to obtain the visualization rendering quality score. The visual fidelity threshold was determined through user surveys and experimental testing, with different threshold ranges corresponding to different types of visualization tasks.

4. The data acquisition and visualization processing system based on big data analysis according to claim 3, characterized in that: The calculation formula for the dynamic threshold adjustment component is as follows: ； In the formula: DHL new The updated data hierarchy threshold is used, VRRQ0 is the target rendering quality score, and REC is used for the target rendering quality score. opt This represents the optimal resource elasticity coefficient. When the updated data hierarchy segmentation threshold DHL new When the value is less than 0.3: the core data range is less than 30%, and only the most critical data is stored in the core storage area; When 0.3 ≤ the updated data hierarchy partitioning threshold DHL new When the value is less than 0.7: 30%-70% of the core data should be stored in the critical storage area; When the updated data hierarchy segmentation threshold DHL new When the value is ≥0.7: the core data range is ≥70%, and most of the data is stored in the core storage area.

5. The data acquisition and visualization processing system based on big data analysis according to claim 4, characterized in that: The system also includes a visualization rendering component, which is configured to perform visualization rendering using an adaptive rendering algorithm based on the visualization rendering quality score calculated by the visualization quality assessment component. The visualization rendering component includes: The adaptive rendering module is configured to dynamically adjust rendering parameters, including resolution, frame rate, and color depth, based on the visual rendering quality score. The visualization type switching module is configured to switch visualization types according to user needs and data types. It adopts a template-based design and supports the rapid generation of different types of visualization results.

6. The data acquisition and visualization processing system based on big data analysis according to claim 1, characterized in that: The resource allocation component, the visual quality assessment component, and the dynamic threshold adjustment component achieve collaborative optimization through a closed-loop feedback mechanism. The dynamic threshold adjustment component adjusts the data hierarchy partitioning threshold based on visualization quality and system resource status. Changes in the data hierarchy partitioning threshold will affect the calculation of the business requirement data and the dynamic resource quantity. The resource allocation component adjusts the resource allocation scheme according to the dynamic resource quantity. The change in the resource allocation scheme affects the resource utilization efficiency, the data loading efficiency, and the calculation of the visualization rendering quality score. The visualization quality assessment component adjusts the visualization rendering parameters based on the visualization rendering quality score. Changes in the visualization rendering parameters affect the visualization quality, system resource status, and the calculation of the updated data hierarchy division threshold.

Images