A big data analysis and display system with dynamic configuration

Abstract

The present invention discloses a dynamically configured big data analysis and display system, relating to the field of big data analysis and display technology. The system comprises a dynamic configuration management module and a data acquisition module. The data acquisition module uses a predefined interface to collect data from multiple heterogeneous data sources. The dynamic configuration management module is based on an event-driven architecture and includes a status monitoring submodule, a condition judgment submodule, and a configuration update submodule. By designing an event-driven dynamic configuration management module, the present invention implements the function of automatically detecting the operating status of each module and triggering configuration updates based on preset thresholds. This solves the problem of traditional systems having fixed configurations and being unable to dynamically adapt to data fluctuations and load changes, enabling system adaptive adjustment and improving overall flexibility and collaborative operation capabilities.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis and display technology, and specifically to a big data analysis and display system with dynamic configuration. Background Art

[0002] With the advent of the big data era, all industries have put forward higher requirements for the real-time, flexibility and intelligence of data analysis and presentation.

[0003] Traditional big data systems generally use static configurations, which make it difficult to cope with dynamically changing data volumes, diverse data types, and user personalized needs. At the same time, there are many bottlenecks in the coordinated scheduling, resource allocation, and linkage display between systems in a distributed environment.

[0004] Patent CN119377687B discloses an innovative display solution system based on big data. The above patent realizes labeled classification in the data acquisition module, optimizes the classification of labels by using the residual diversification integration method, combines deep classifiers and shallow classifiers to form a diversified model structure, and introduces a generative adversarial network to generate tail label samples; introduces the C-WOA optimization algorithm in the training of the diversified integration model, adopts an adaptive parameter adjustment mechanism, and dynamically optimizes the model parameter search range and step size; constructs a graph convolutional network model in the data analysis module to analyze user behavior data and data packets, introduces a dynamic graph structure, and forms a high-dimensional graph structure distribution through the variational inference method.

[0005] The above patent optimizes the model's ability to process time series data, but the system disclosed in the above patent adopts a static configuration and is difficult to cope with dynamically changing data.

[0006] To this end, this application proposes a big data analysis and display system for dynamic configuration based on event-driven configuration updates. Summary of the Invention

[0007] The purpose of the present invention is to provide a big data analysis and display system with a dynamic configuration to solve the technical problem raised in the above background technology that it is difficult to cope with dynamically changing data and personalized needs.

[0008] To achieve the above objectives, the present invention provides the following technical solutions: a big data analysis and display system with dynamic configuration, comprising a dynamic configuration management module and a data acquisition module. The data acquisition module uses a predefined interface to collect data from multiple heterogeneous data sources. The dynamic configuration management module is based on an event-driven architecture and includes a status monitoring submodule, a condition judgment submodule, and a configuration update submodule.

[0009] The state monitoring submodule periodically collects real-time states from the data acquisition module and downstream modules, and passes the collected data to the condition judgment submodule;

[0010] The condition judgment submodule relies on the rule library to judge the need for updating the configuration, and sends the updated parameters in the form of a microservice interface through the configuration update submodule.

[0011] Preferably, the data acquisition module is internally provided with a data format conversion unit and a data buffer storage unit;

[0012] The data format conversion unit can support automatic conversion between JSON, XML and CSV formats;

[0013] The data buffer storage unit uses memory database and local cache technology to temporarily store original data, and transmits it to the dynamic configuration management module after identifying the data type through predefined data tags.

[0014] Preferably, the system includes a distributed data processing and analysis module, which relies on a distributed computing platform to implement data preprocessing, parsing and data mining, and internally integrates a deep reinforcement learning intelligent agent;

[0015] The intelligent agent simultaneously collects data passed in by the data acquisition module and configuration parameters issued by the dynamic configuration management module, generates parameters for data segmentation, task scheduling and resource allocation based on the internal state prediction model, and feeds them back to the dynamic configuration management module.

[0016] Preferably, the system includes a data display module, which is implemented using a front-end and back-end separation architecture. The front-end renders charts based on the React framework, and uses WebSocket to receive data from the distributed data processing and analysis module and display parameters transmitted by the dynamic configuration management module in real time, thereby realizing chart linkage and adaptive layout.

[0017] Preferably, the status monitoring submodule in the dynamic configuration management module periodically collects operating parameters from the data acquisition module, the distributed data processing and analysis module, and the data display module. The operating parameters include packet rate, delay, and load status, and are transmitted to the condition judgment submodule in a unified format. The condition judgment submodule has a built-in rule library based on multi-layer rule judgment logic. The rule library stores the critical state thresholds of each module and the corresponding configuration parameter sets for real-time call by the subsequent configuration update submodule.

[0018] Preferably, the configuration update submodule in the dynamic configuration management module realizes data interaction with each downstream module through a microservice interface, and the configuration update submodule includes a configuration instruction encoding unit and a version control unit;

[0019] The configuration instruction encoding unit encapsulates the updated parameters in a standardized manner and sends the instructions to the distributed data processing and analysis module and the data display module through the message queue. At the same time, after receiving feedback information, it forms a two-way communication with the status monitoring submodule.

[0020] Preferably, a deep reinforcement learning intelligent agent is provided inside the distributed data processing and analysis module, and the intelligent agent includes a historical state collection unit, a real-time state update unit and a strategy generation unit;

[0021] The historical status collection unit collects historical data and status information from the data collection module and the dynamic configuration management module;

[0022] The real-time status update unit continuously receives the current processing status;

[0023] The strategy generation unit generates specific data segmentation, task scheduling and resource allocation parameters based on the set neural network structure, and feeds the generated parameters back to the dynamic configuration management module for further adjustment of downstream processing.

[0024] Preferably, the distributed data processing and analysis module is further provided with a data prediction linkage submodule, which adopts a Transformer or LSTM network to realize the prediction of time series data features. The prediction results and the current data flow characteristics are transmitted to the dynamic configuration management module through a predefined interface to update the trigger conditions in the rule base and realize early warning of the data flow status.

[0025] Preferably, the data display module includes a data parsing submodule and a multi-view rendering submodule;

[0026] The data parsing submodule uses format standardization logic to parse the output data from the distributed data processing and analysis module, and packages the data according to the display parameters issued by the dynamic configuration management module;

[0027] The multi-view rendering submodule parses the parameter data according to the information linkage rules, renders various visual charts through SVG, and supports linkage calls and data synchronization between charts.

[0028] Preferably, each module is connected by a communication unit of a microservice architecture, which includes a RESful API call interface, a gRPC communication interface, and a message queue interface based on RabbitMQ. Each module is embedded with a unified communication protocol parser to achieve unified encoding, decoding, and verification of data packet formats, ensuring interoperability and synchronization of data flows and control instruction transmission between modules.

[0029] The system adopts a containerized deployment structure. All modules are encapsulated in independent Docker containers and orchestrated and managed through a Kubernetes cluster. The Kubernetes cluster is equipped with an automatic scaling unit, a service discovery module, and a node health detection unit.

[0030] The service discovery module works in conjunction with the dynamic configuration management module to periodically detect the status of each container and feed the container status data back to the dynamic configuration management module to ensure consistent logical scheduling within the system.

[0031] A unified state synchronization control platform is set up inside the system. It receives status and log information from the data acquisition module, dynamic configuration management module, distributed data processing and analysis module, and data display module through a dedicated data acquisition interface. It performs real-time unified monitoring based on Prometheus's standardized data aggregation mechanism, and feeds the aggregation results back to the dynamic configuration management module to achieve logical linkage, verification and closed-loop update of the configuration parameters and status information of the entire system.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention utilizes an event-driven dynamic configuration management module to automatically detect the operating status of each module and trigger configuration updates based on preset thresholds. This solves the problem of traditional systems with fixed configurations and inability to dynamically adapt to data fluctuations and load changes, enabling system adaptive adjustment and improving overall flexibility and collaborative operation capabilities.

[0034] 2. This invention optimizes data processing processes and resource scheduling decisions by designing a deep reinforcement learning intelligent agent in a distributed data analysis and processing module. This addresses the lack of self-learning capabilities in traditional systems, making it difficult to optimize scheduling for real-time changing data loads. It improves processing efficiency and resource utilization, and reduces system response latency.

[0035] 3. The present invention implements a data prediction linkage submodule to provide early warning of data flow state changes, adjust configuration parameters to proactively address data peaks, and address data surges and processing delays encountered during real-time data processing. This ensures stable system operation during peak periods and further improves data processing robustness.

[0036] 4. The present invention is designed with a unified communication and monitoring platform based on microservices and container orchestration to achieve efficient, standard and real-time data packet transmission and status feedback functions between modules, solving the problems of inconsistent communication protocols, scattered status information and difficult management between different modules in a distributed system, improving the system's interoperability, synchronization and overall stability, and facilitating automatic expansion and contraction and fault self-healing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the system architecture of the present invention;

[0038] Figure 2 Schematic diagram of the system workflow of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In the description of the present invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," "the other end," and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limiting the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "provided with," "connected," etc., should be understood in a broad sense. For example, "connected" may refer to a fixed connection, a detachable connection, or an integral connection; it may refer to a mechanical connection or an electrical connection; it may refer to a direct connection or an indirect connection through an intermediate medium; it may refer to internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0042] See also Figure 1 and Figure 2 The present invention provides an embodiment of a big data analysis and display system with dynamic configuration, comprising a dynamic configuration management module and a data acquisition module. The data acquisition module uses a predefined interface to collect data from multiple heterogeneous data sources. The dynamic configuration management module is based on an event-driven architecture and includes a status monitoring submodule, a condition judgment submodule, and a configuration update submodule.

[0043] The state monitoring submodule periodically collects real-time states from the data acquisition module and downstream modules, and passes the collected data to the condition judgment submodule;

[0044] The condition judgment submodule relies on the rule base to judge the need for updating the configuration, and sends the updated parameters in the form of a microservice interface through the configuration update submodule;

[0045] The data acquisition module is internally provided with a data format conversion unit and a data buffer storage unit;

[0046] The data format conversion unit can support automatic conversion between JSON, XML and CSV formats;

[0047] The data buffer storage unit uses memory database and local cache technology to temporarily store raw data, and transmits it to the dynamic configuration management module after identifying the data type through predefined data tags;

[0048] Furthermore, the system sets up multiple predefined interfaces for connecting to different types of data sources. Assume that the system is connected to three data sources at the same time: data source A outputs in JSON format, data source B in CSV format, and data source C in XML format. Each data source sends data to the data acquisition module through its own predefined interface, such as a RESTful interface. Inside the data acquisition module, all received data first enters the data format conversion unit. When the JSON data sent by data source A enters, the data format conversion unit identifies the format and directly releases it. When the CSV data sent by data source B arrives, the data format conversion unit parses the CSV content into a standard JSON format according to predefined conversion rules. When the XML data sent by data source C arrives, the XML content is also converted into a JSON format consistent with other data according to predefined rules. Through this conversion, all data is output in a unified format, providing format consistency for subsequent processing. The converted data is then stored in the data buffer storage unit. The system uses an in-memory database and local cache technology to temporarily store the original data. During storage, the system appends a predefined data tag to the data packet. The tag identifies the source and data type of the data to ensure that subsequent modules can correctly distinguish the data type when processing.

[0049] The status monitoring submodule first collects the data reception rate, the rate after data format conversion, and the current number of data entries in the buffer storage unit from the data acquisition module; at the same time, through the interface communication with the downstream distributed data processing module, it collects the real-time processing status of the downstream module, such as the data processing queue depth or processing delay index. The collected status information is packaged in a unified format and passed to the condition judgment submodule; when the data transmitted by the status monitoring submodule indicates that a certain parameter exceeds the preset threshold, the condition judgment submodule compares it through the internal condition judgment logic. After judgment, if there is a matching update condition in the rule base, the condition judgment submodule generates a corresponding update request information and passes the judgment result to the configuration update submodule; the configuration update submodule constructs a new configuration parameter package, which specifically includes the working parameters of each module, such as the data buffer size, data acquisition rate, etc., and version encodes the parameter package. Through the system's built-in microservice interface, the parameter package is sent to the data acquisition module and other downstream modules connected to it in a message format to ensure that each module in the system receives a unified configuration update instruction.

[0050] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: a dynamically configured big data analysis and display system, the system includes a distributed data processing and analysis module, relies on a distributed computing platform to implement data preprocessing, parsing and data mining, and internally integrates a deep reinforcement learning intelligent agent;

[0051] The intelligent agent simultaneously collects data from the data acquisition module and configuration parameters from the dynamic configuration management module. Based on the internal state prediction model, it generates parameters for data segmentation, task scheduling, and resource allocation, and feeds them back to the dynamic configuration management module.

[0052] The distributed data processing and analysis module is internally provided with a deep reinforcement learning intelligent agent, which includes a historical state collection unit, a real-time state update unit and a strategy generation unit;

[0053] The historical status collection unit collects historical data and status information from the data collection module and the dynamic configuration management module;

[0054] The real-time status update unit continuously receives the current processing status;

[0055] The strategy generation unit generates specific data segmentation, task scheduling, and resource allocation parameters based on the set neural network structure, and feeds the generated parameters back to the dynamic configuration management module for further adjustment of downstream processing;

[0056] The distributed data processing and analysis module is also provided with a data prediction linkage submodule, which uses a Transformer or LSTM network to predict the characteristics of time series data. The prediction results and the current data flow characteristics are transmitted to the dynamic configuration management module through a predefined interface to update the trigger conditions in the rule base and realize early warning of the data flow status.

[0057] Furthermore, the historical status acquisition unit collects data including the timestamp, data type identifier, data volume, anomaly mark, etc. output by the data acquisition module. At the same time, it obtains the previously issued configuration parameters and status feedback from the dynamic configuration management module. The collected data is stored in the internal historical status database in a time series manner for subsequent neural network training and strategy generation. The real-time status update unit subscribes to the status update events of the distributed processing platform, such as the processing progress, delay, queue depth and resource utilization of each task. The received status data is transmitted to the strategy generation unit inside the intelligent agent through the internal bus in a standardized JSON format. The real-time status data is matched with the historical data on the timeline. Provide current context information to the neural network; the strategy generation unit uses a preset neural network structure, such as a convolutional neural network combined with a long short-term memory network, to predict the current state and generate specific operation parameters. The input end of the neural network model receives joint data from the historical state acquisition unit and the real-time state update unit. During the training process, the model uses supervised learning or reinforcement learning frameworks to build a state-strategy correspondence. The training goal is to output specific parameters such as data segmentation granularity, task scheduling priority, and resource allocation ratio. After the output parameters form a policy data packet, after internal coding and version confirmation, they are fed back to the dynamic configuration management module through a predefined microservice interface for the latter to update the configuration of downstream processing;

[0058] The data prediction linkage submodule uses the Transformer network as the time series data feature prediction model. The model training data comes from the time series data and key status indicators recorded by the historical state acquisition unit. The preprocessing process includes data normalization, noise filtering and time window segmentation to ensure that the input data meets the requirements of the neural network. After training is completed, the model is saved as a standardized model file for online reasoning. The sliding window method is used to perform real-time prediction of the input data sequence to generate predicted values ​​of key indicators in the future time series. The prediction results are encapsulated into a data packet together with the real-time data stream characteristics and passed to the dynamic configuration management module through a predefined interface. The dynamic configuration management module updates the trigger conditions in its internal rule base based on the received prediction data, and includes possible data states in the future into the monitoring scope in advance.

[0059] See also Figure 1 and Figure 2The present invention provides an embodiment of a dynamically configured big data analysis and display system, comprising a data display module, implemented using a front-end and back-end separation architecture, wherein the front-end renders charts based on a React framework, and utilizes WebSocket to receive data from a distributed data processing and analysis module and display parameters transmitted by a dynamic configuration management module in real time, thereby achieving chart linkage and adaptive layout.

[0060] The data display module includes a data parsing submodule and a multi-view rendering submodule;

[0061] The data parsing submodule uses format standardization logic to parse the output data from the distributed data processing and analysis module, and packages the data according to the display parameters issued by the dynamic configuration management module;

[0062] The multi-view rendering submodule parses parameter data according to information linkage rules, renders various visual charts through SVG, and supports linkage calls and data synchronization between charts;

[0063] Furthermore, after the front-end receives the data packet, the data parsing submodule first performs data preprocessing. This submodule contains format standardization logic. Its processing process is as follows: the structure of the "data part" in the data packet is detected and compared with the predefined format template; the raw data is converted into the system's internal standard format according to predefined rules, such as renaming each data item to a unified field name, converting date strings to timestamps, and normalizing various measurement values; combined with the parameters of the "configuration part", such as the specified chart data range, partitioning standards, data filtering conditions, etc., the data is repackaged to generate the structured data objects required for rendering;

[0064] After receiving the data package object from the data parsing submodule, the multi-view rendering submodule first performs information linkage rule parsing. It preloads the display parameters and linkage rules issued by the dynamic scheduling management module. These rules include the linkage logic between charts and the specific attribute definitions of each chart. It then performs rule matching on the parsed data objects to determine which chart each data field should be mapped to, and parses the drawing parameters of each chart.

[0065] After the parsing is completed, the multi-view rendering sub-module uses SVG technology to generate visual charts based on the matched parameters, calls the SVG rendering library inside the React component to draw the corresponding charts based on the parsed data, and calculates the display position, size, and spacing of each view according to the layout parameters defined in the configuration file to ensure adaptive layout at different resolutions. Event listeners are embedded in the React component. When the user interacts with any chart, such as clicking, zooming, or hovering the mouse, the event information is passed to other chart components through the internal event scheduling mechanism to achieve linked calls and data synchronization. For example, when the user selects a time interval in the line chart, other charts automatically refresh the data statistics within that time period.

[0066] See also Figure 1 and Figure 2 The present invention provides an embodiment of a big data analysis and display system with dynamic configuration. The state monitoring submodule in the dynamic configuration management module periodically collects operating parameters from the data acquisition module, the distributed data processing and analysis module, and the data display module. The operating parameters include packet rate, delay, and load status, and are transmitted to the condition judgment submodule in a unified format. The condition judgment submodule has a built-in rule library based on multi-layer rule judgment logic. The rule library stores the critical state threshold value and corresponding configuration parameter set of each module for real-time call by the subsequent configuration update submodule.

[0067] The configuration update submodule in the dynamic configuration management module implements data interaction with each downstream module through the microservice interface. The configuration update submodule includes a configuration instruction encoding unit and a version control unit;

[0068] The configuration instruction encoding unit encapsulates the updated parameters in a standardized manner and sends the instructions to the distributed data processing and analysis module and the data display module through the message queue. At the same time, after receiving feedback information, it forms a two-way communication with the status monitoring submodule;

[0069] Furthermore, the conditional judgment submodule has a built-in multi-layer rule base, which presets the critical state thresholds of each module and the corresponding configuration parameter sets. For example: the data acquisition module rules, when the data packet rate exceeds the preset upper limit, such as 1300 packets / second or the delay exceeds 50 milliseconds, the preset update parameters are "reducing the acquisition frequency" or "increasing the buffer zone"; the distributed data processing module rules, when the processing delay exceeds 70 milliseconds or the node load CPU>80% reaches a critical state, the preset update parameters are "adjusting the task scheduling strategy" or "dynamically increasing or decreasing computing resources"; the data display module rules, when the chart update delay or linkage response delay exceeds the preset value, the preset update parameters are "optimizing the data parsing rate" or "adjusting the display refresh interval"; after the conditional judgment submodule receives the unified format data transmitted by the status monitoring submodule, it compares it according to the preset multi-layer rule judgment logic. If any indicator in the data exceeds its preset threshold, the corresponding configuration parameter set is selected to generate an update request record. For example, in data processing, if the "processing delay" reaches 80 milliseconds and the CPU load is 82%, the condition judgment submodule will determine the update operation parameters to be "reducing data segmentation granularity" or "adjusting scheduling queue priority", etc., and form this information into a standardized update request data packet and pass it to the configuration update submodule;

[0070] The configuration update submodule is responsible for converting the update request generated by the condition judgment submodule into standardized configuration instructions, and interacting with downstream modules through the microservice interface. The specific process is as follows: Configuration instruction encoding unit, parameter standardization: the parameter set passed in by the conditional judgment submodule is encapsulated to generate an instruction data packet in a unified format, with an instruction version number attached; message queue distribution, the configuration update submodule sends the standardized instruction data packet to the designated downstream distributed data processing and analysis module and data display module through a predefined message queue to ensure that each module can receive the new configuration instructions in a timely manner; version control unit, to ensure the consistency and traceability of instructions, the version control unit records the version information and sending time of each instruction update, and compares it with the feedback information of each downstream module to confirm that the instruction has been correctly accepted and executed; two-way feedback interaction, after receiving the configuration instruction, the downstream module executes the configuration update and feedbacks the update execution status through the same message queue or directly calls the microservice interface, such as "update successful" or "partial parameter update failed". After receiving the feedback information, the configuration update submodule passes the feedback data to the status monitoring submodule, forming a two-way data interaction loop, so that the system status can be re-sampled and rule-judged later, realizing closed-loop monitoring and dynamic configuration update.

[0071] See also Figure 1 and Figure 2The present invention provides an embodiment of a dynamically configured big data analysis and display system, wherein each module is connected using a communication unit of a microservice architecture, the communication unit including a RESful API call interface, a gRPC communication interface, and a RabbitMQ-based message queue interface. Each module is embedded with a unified communication protocol parser to achieve unified encoding, decoding, and verification of data packet formats, ensuring interoperability and synchronization of data flows and control instruction transmission between modules.

[0072] The system adopts a containerized deployment structure. All modules are encapsulated in independent Docker containers and orchestrated and managed through a Kubernetes cluster. The Kubernetes cluster is equipped with an automatic scaling unit, a service discovery module, and a node health detection unit.

[0073] The service discovery module works in conjunction with the dynamic configuration management module to periodically detect the status of each container and feed the container status data back to the dynamic configuration management module to ensure consistent logical scheduling within the system.

[0074] The system has a unified state synchronization control platform, which receives status and log information from the data acquisition module, dynamic configuration management module, distributed data processing and analysis module, and data display module through a dedicated data acquisition interface. It performs real-time unified monitoring based on Prometheus's standardized data aggregation mechanism and feeds the aggregation results back to the dynamic configuration management module to achieve logical linkage, verification, and closed-loop update of system-wide configuration parameters and status information.

[0075] Furthermore, each module in the system, such as the data collection module, dynamic configuration management module, distributed data processing and analysis module, and data display module, are encapsulated in independent Docker containers to ensure isolation of the operating environment and unified dependencies between modules. Each container image contains a pre-defined communication protocol parser, built-in log collection, and health monitoring functions, so that the container can provide standardized service interfaces to the outside world. Kubernetes automatically adjusts the number of container replicas for each module according to the container load, and uses the Horizontal Pod Autoscaler to ensure that resources can be dynamically scheduled in high concurrency or emergency situations. The Kubernetes cluster has built-in service discovery mechanisms such as CoreDNS, and each module calls each other through service names. The service discovery module works in conjunction with the dynamic configuration management module to periodically detect the operating status of all containers, such as online status, response time, and health check results, and feeds these container status data back to the dynamic configuration management module. Kubernetes regularly checks the health of each node to ensure a stable container deployment environment. Abnormal nodes will be isolated or rebuilt to maintain consistency in the logical scheduling and configuration parameters of the entire system.

[0076] A unified state synchronization control platform is set up inside the system. Each module sends status information and log data to the platform through a dedicated interface. The status data includes the operating parameters, response delay, packet rate, exception log, etc. of each module. After the log data is converted into a unified format, it forms a standardized data packet to facilitate aggregation and subsequent analysis. The state synchronization control platform uses monitoring tools such as Prometheus and standardized collection protocols such as the PromQL query language to regularly capture and aggregate the status data of each module. The aggregated monitoring data is fed back to the dynamic configuration management module through a dedicated data interface, enabling it to verify the configuration parameters of each module based on the global status information, and then trigger the necessary configuration updates to form a complete closed-loop management.

[0077] Working Principle: The system first collects raw data from multiple heterogeneous data sources through predefined interfaces. The data format conversion unit converts JSON, XML, CSV, and other formats into a standard format, and then uses the data buffer storage unit for temporary caching. The dynamic configuration management module is activated based on an event-driven architecture. Its built-in status monitoring submodule regularly collects real-time operating parameters from the data acquisition module and downstream modules, and transmits them to the condition judgment submodule in a unified format to determine whether the system configuration needs to be adjusted.

[0078] The collected data enters the distributed data processing and analysis module, which relies on big data computing platforms such as Spark and Flink to preprocess, analyze, and conduct deep data mining. The deep reinforcement learning intelligent agent integrated within the module uses the historical state collection unit and the real-time state update unit to generate specific parameters for data segmentation, task scheduling, and resource allocation based on the preset neural network model, and feeds the generated strategy back to the dynamic configuration management module. The data prediction linkage submodule uses the Transformer or LSTM model to perform time series predictions on future data flow characteristics and transmits them in advance to the dynamic configuration management module for updating preset rules.

[0079] After data processing is completed, the processing results are passed to the data display module through the distributed data processing module. The front-end adopts a front-end and back-end separation architecture based on React and WebSocket real-time communication to achieve chart linkage, adaptive layout and multi-view display; at the same time, the data display module feeds back user interaction data and display status to the dynamic configuration management module, forming a closed loop with the operating parameters between each module, and realizing unified communication and status synchronization of the entire system through microservice interfaces, gRPC and RabbitMQ-based message queues, ensuring continuous optimization of full-link parameters and dynamic configuration updates.

[0080] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A big data analysis and display system with dynamic configuration, including a dynamic configuration management module and a data acquisition module, characterized by: The data acquisition module uses a predefined interface to collect data from multiple heterogeneous data sources. The dynamic configuration management module is based on an event-driven architecture and has a status monitoring submodule, a condition judgment submodule, and a configuration update submodule. The state monitoring submodule periodically collects real-time states from the data acquisition module and downstream modules, and passes the collected data to the condition judgment submodule; The condition judgment submodule relies on the rule base to judge the need for updating the configuration, and sends the updated parameters in the form of a microservice interface through the configuration update submodule; The system includes a distributed data processing and analysis module, which relies on a distributed computing platform to achieve data preprocessing, analysis and data mining; The distributed data processing and analysis module is internally provided with a deep reinforcement learning intelligent agent, which includes a historical state collection unit, a real-time state update unit and a strategy generation unit; The historical status collection unit collects historical data and status information from the data collection module and the dynamic configuration management module; The real-time status update unit continuously receives the current processing status; The strategy generation unit generates specific data segmentation, task scheduling, and resource allocation parameters based on the set neural network structure, and feeds the generated parameters back to the dynamic configuration management module for further adjustment of downstream processing; The distributed data processing and analysis module is also provided with a data prediction linkage submodule, which uses a Transformer or LSTM network to realize the prediction of time series data features. The prediction results and the current data flow characteristics are transmitted to the dynamic configuration management module through a predefined interface to update the trigger conditions in the rule base and realize early warning of data flow status.

2. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The data acquisition module is internally provided with a data format conversion unit and a data buffer storage unit; The data format conversion unit can support automatic conversion between JSON, XML and CSV formats; The data buffer storage unit uses memory database and local cache technology to temporarily store original data, and transmits it to the dynamic configuration management module after identifying the data type through predefined data tags.

3. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The system includes a data display module, which is implemented using a front-end and back-end separation architecture. The front-end renders charts based on the React framework and uses WebSocket to receive data from the distributed data processing and analysis module and display parameters transmitted by the dynamic configuration management module in real time to achieve chart linkage and adaptive layout.

4. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The status monitoring submodule in the dynamic configuration management module periodically collects operating parameters from the data acquisition module, the distributed data processing and analysis module, and the data display module. The operating parameters include packet rate, delay, and load status, and are passed to the condition judgment submodule in a unified format. The condition judgment submodule has a built-in rule library based on multi-layer rule judgment logic. The rule library stores the critical state thresholds of each module and the corresponding configuration parameter sets for real-time call by the subsequent configuration update submodule.

5. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The configuration update submodule in the dynamic configuration management module implements data interaction with each downstream module through the microservice interface. The configuration update submodule includes a configuration instruction encoding unit and a version control unit; The configuration instruction encoding unit encapsulates the updated parameters in a standardized manner and sends the instructions to the distributed data processing and analysis module and the data display module through the message queue. At the same time, after receiving feedback information, it forms a two-way communication with the status monitoring submodule.

6. The big data analysis and display system with dynamic configuration according to claim 3, characterized in that: The data display module includes a data parsing submodule and a multi-view rendering submodule; The data parsing submodule uses format standardization logic to parse the output data from the distributed data processing and analysis module, and packages the data according to the display parameters issued by the dynamic configuration management module; The multi-view rendering submodule parses the parameter data according to the information linkage rules, renders various visual charts through SVG, and supports linkage calls and data synchronization between charts.

7. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: Each module is connected using a microservices-based communication unit, which includes a RESful API call interface, a gRPC communication interface, and a RabbitMQ-based message queue interface. Each module is embedded with a unified communication protocol parser to achieve unified encoding, decoding, and verification of data packet formats, ensuring interoperability and synchronization of data flows and control instruction transmission between modules. The system adopts a containerized deployment structure. All modules are encapsulated in independent Docker containers and orchestrated and managed through a Kubernetes cluster. The Kubernetes cluster is equipped with an automatic scaling unit, a service discovery module, and a node health detection unit. The service discovery module works in conjunction with the dynamic configuration management module to periodically detect the status of each container and feed the container status data back to the dynamic configuration management module to ensure consistent logical scheduling within the system. A unified state synchronization control platform is set up inside the system. It receives status and log information from the data acquisition module, dynamic configuration management module, distributed data processing and analysis module, and data display module through a dedicated data acquisition interface. It performs real-time unified monitoring based on Prometheus's standardized data aggregation mechanism, and feeds the aggregation results back to the dynamic configuration management module to achieve logical linkage, verification and closed-loop update of the configuration parameters and status information of the entire system.

Images