Big data analysis and display system with dynamic configuration

Abstract

The invention discloses a big data analysis and display system with dynamic configuration, and relates to the technical field of big data analysis and display, the system comprises a dynamic configuration management module and a data acquisition module, the data acquisition module adopts a predefined interface, and is used for acquiring data from a plurality of heterogeneous data sources; the dynamic configuration management module is based on an event-driven architecture, and is internally provided with a state monitoring sub-module, a condition judgment sub-module and a configuration updating sub-module. By designing the event-driven dynamic configuration management module, the functions of automatically detecting the running state of each module and triggering configuration updating according to the preset threshold are realized, the problems that a traditional system is fixed in configuration and cannot dynamically adapt to data fluctuation and load change are solved, self-adaptive adjustment of the system is realized, and the service life of the system is prolonged. And the overall flexibility and the collaborative operation capability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis and display, and in particular to a big data analysis and display system with dynamic configuration. Background Art

[0002] With the advent of the big data era, all industries and sectors have put forward higher requirements for the real-time, flexibility and intelligence of data analysis and presentation.

[0003] Traditional big data systems generally use static configurations, which make it difficult to cope with dynamically changing data volumes, diverse data types, and personalized user needs. At the same time, there are many bottlenecks in the coordinated scheduling, resource allocation, and linkage display between systems in a distributed environment.

[0004] Patent CN119377687B discloses an innovative display solution system based on big data. The above patent realizes labeled classification in the data acquisition module, optimizes the classification of labels by using the residual diversified integration method, combines deep classifiers and shallow classifiers to form a diversified model structure, and introduces a generative adversarial network to generate tail label samples; introduces the C-WOA optimization algorithm in the training of the diversified integration model, adopts an adaptive parameter adjustment mechanism, and dynamically optimizes the model parameter search range and step size; constructs a graph convolutional network model in the data analysis module to analyze user behavior data and data packets, introduces a dynamic graph structure, and forms a high-dimensional graph structure distribution through variational inference methods.

[0005] The above patent optimizes the model's ability to process time series data, but the system disclosed in the above patent adopts a static configuration and is difficult to cope with dynamically changing data.

[0006] To this end, the present application proposes a big data analysis and display system for dynamic configuration based on event-driven configuration updates. Summary of the invention

[0007] The purpose of the present invention is to provide a big data analysis and display system with a dynamic configuration to solve the technical problem of being difficult to cope with dynamically changing data and personalized needs raised in the above-mentioned background technology.

[0008] To achieve the above-mentioned object, the present invention provides the following technical solutions: a big data analysis and display system with dynamic configuration, comprising a dynamic configuration management module and a data acquisition module, wherein the data acquisition module adopts a predefined interface to collect data from multiple heterogeneous data sources, and the dynamic configuration management module is based on an event-driven architecture and has a status monitoring submodule, a condition judgment submodule and a configuration update submodule; The state monitoring submodule periodically collects real-time states from the data acquisition module and downstream modules, and transmits the collected data to the condition judgment submodule; The condition judgment sub-module relies on the rule library to judge the need for configuration update, and sends the updated parameters to the downstream through the configuration update sub-module in the form of a microservice interface.

[0009] Preferably, a data format conversion unit and a data buffer storage unit are provided inside the data acquisition module; The data format conversion unit can support automatic conversion between JSON, XML, and CSV formats; The data buffer storage unit uses an in-memory database and local cache technology to temporarily store the original data, and transmits the data to the dynamic configuration management module after identifying the data type through predefined data tags.

[0010] Preferably, the system includes a distributed data processing and analysis module, which relies on a distributed computing platform to implement data preprocessing, parsing, and data mining, and integrates a deep reinforcement learning intelligent agent internally; The intelligent agent simultaneously collects the data input from the data acquisition module and the configuration parameters sent by the dynamic configuration management module, generates parameters for data segmentation, task scheduling, and resource allocation based on the internal state prediction model, and feeds them back to the dynamic configuration management module.

[0011] Preferably, the system includes a data display module, which is implemented using a front-end and back-end separation architecture. The front-end renders charts based on the React framework, and uses WebSocket to receive data from the distributed data processing and analysis module and display parameters passed by the dynamic configuration management module in real time to achieve chart linkage and adaptive layout.

[0012] Preferably, the status monitoring sub-module in the dynamic configuration management module periodically collects operation parameters from the data acquisition module, the distributed data processing and analysis module, and the data display module. The operation parameters include data packet rate, latency, and load status, and are transmitted to the condition judgment sub-module in a unified format. The condition judgment sub-module has a rule library built with a multi-layer rule judgment logic, and the rule library stores the critical state thresholds of each module and the corresponding configuration parameter sets for subsequent real-time invocation by the configuration update sub-module.

[0013] Preferably, the configuration update sub-module in the dynamic configuration management module realizes data interaction with each downstream module through a microservice interface. The configuration update sub-module includes a configuration instruction encoding unit and a version control unit; The configuration instruction encoding unit standardizes and encapsulates the updated parameters, and sends the instructions to the distributed data processing and analysis module and the data display module through a message queue. At the same time, after receiving the feedback information, it forms a two-way communication with the status monitoring sub-module.

[0014] Preferably, a deep reinforcement learning intelligent agent is provided inside the distributed data processing and analysis module, and the intelligent agent includes a historical state acquisition unit, a real-time state update unit, and a policy generation unit; The historical state acquisition unit acquires historical data and status information from the data acquisition module and the dynamic configuration management module; The real-time state update unit continuously receives the current processing status; The policy generation unit generates specific data segmentation, task scheduling, and resource allocation parameters based on the set neural network structure, and feeds the generated parameters back to the dynamic configuration management module for further adjustment of downstream processing.

[0015] Preferably, a data prediction linkage sub-module is further provided in the distributed data processing and analysis module. The data prediction linkage sub-module uses a Transformer or LSTM network to implement time-series data feature prediction. The prediction result and the current data stream characteristics are transmitted to the dynamic configuration management module through a predefined interface, which is used to update the trigger conditions in the rule library to achieve early warning of the data stream state.

[0016] Preferably, the data display module includes a data parsing sub-module and a multi-view rendering sub-module; The data parsing sub-module parses the output data from the distributed data processing and analysis module using format standardization logic, and packs the data according to the display parameters issued by the dynamic configuration management module; The multi-view rendering sub-module parses the parameter data according to the information linkage rules, and then renders various visualization charts through SVG, and supports the linkage call and data synchronization between charts.

[0017] Preferably, the communication unit of the microservices architecture is used to connect between modules. The communication unit includes a RESful API call interface, a gRPC communication interface, and a message queue interface based on RabbitMQ. Each module is embedded with a unified communication protocol parser to realize unified encoding, decoding, and verification of the data packet format, ensuring the interoperability and synchronization of data stream and control instruction transmission between modules; The system adopts a containerized deployment structure. All modules are encapsulated in independent Docker containers and are orchestrated and managed through a Kubernetes cluster. The Kubernetes cluster is configured with an automatic scaling unit, a service discovery module, and a node health detection unit; The service discovery module is linked with the dynamic configuration management module to periodically detect the status of each container and feed the container status data back to the dynamic configuration management module to ensure the consistency of internal logic scheduling of the system; There is a unified status synchronization control platform inside the system, which receives the status and log information from the data acquisition module, dynamic configuration management module, distributed data processing and analysis module, and data display module through a dedicated data acquisition interface, performs real-time unified monitoring based on the standardized data aggregation mechanism of Prometheus, and feeds back the aggregation result to the dynamic configuration management module to achieve the logical linkage, verification, and closed-loop update of the configuration parameters and status information of the entire system.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By designing an event-driven dynamic configuration management module, the present invention realizes the function of automatically detecting the operating status of each module and triggering configuration updates according to preset thresholds, solves the problem that the configuration of traditional systems is fixed and cannot dynamically adapt to data fluctuations and load changes, realizes the self-adaptive adjustment of the system, and improves the overall flexibility and collaborative operation ability. 2. By designing a deep reinforcement learning intelligent agent in the distributed data analysis and processing module, the present invention realizes the function of optimizing the data processing process and resource scheduling decision-making, solves the problem that traditional systems lack self-learning ability and are difficult to optimize scheduling for real-time changing data loads, improves the processing efficiency and resource utilization rate, and reduces the system response delay. 3. By designing a data prediction linkage sub-module, the present invention realizes the function of warning the change of data flow status, adjusts configuration parameters to cope with data peaks in advance, solves the problems of sudden data increase and processing delay encountered in the real-time data processing process, ensures the stable operation of the system during peak periods, and further improves the robustness of data processing. 4. By designing a unified communication and monitoring platform based on microservices and container orchestration, the present invention realizes the functions of efficient, standard, and real-time data packet transmission and status feedback between modules, solves the problems of inconsistent communication protocols, scattered status information, and difficult management between different modules in a distributed system, improves the interoperability, synchronization, and overall stability of the system, and facilitates automatic scaling and fault self-healing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of the system architecture of the present invention; Figure 2 is a schematic diagram of the system working process of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0022] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: a big data analysis and display system with dynamic configuration, comprising a dynamic configuration management module and a data acquisition module, the data acquisition module adopts a predefined interface for collecting data from multiple heterogeneous data sources, the dynamic configuration management module is based on an event-driven architecture, and has a status monitoring submodule, a condition judgment submodule and a configuration update submodule; The state monitoring submodule periodically collects real-time states from the data acquisition module and downstream modules, and transmits the collected data to the condition judgment submodule; The condition judgment submodule relies on the rule base to judge the need for updating the configuration, and sends the updated parameters in the form of a microservice interface through the configuration update submodule; The data acquisition module is internally provided with a data format conversion unit and a data buffer storage unit; The data format conversion unit can support automatic conversion between JSON, XML and CSV formats; The data buffer storage unit uses memory database and local cache technology to temporarily store raw data, and transmits it to the dynamic configuration management module after identifying the data type through predefined data tags; Furthermore, the system sets multiple predefined interfaces for connecting different types of data sources; assuming that the system is connected to three data sources at the same time: the output format of data source A is JSON, the output format of data source B is CSV, and the output format of data source C is XML. Each data source sends data to the data acquisition module through its own predefined interface, such as the RESTful interface; inside the data acquisition module, all received data first enters the data format conversion unit; when the JSON data sent by data source A enters, the data format conversion unit directly releases it after identifying the format. When the CSV data sent by data source B arrives, the data format conversion unit parses the CSV content into a standard JSON format according to the predefined conversion rules. When the XML data sent by data source C arrives, the XML content is also converted into a JSON format consistent with other data according to the predefined rules. Through this conversion, all data are output in a unified format to provide format consistency for subsequent processing; the converted data is then stored in the data buffer storage unit. The system uses an in-memory database and local cache technology to achieve temporary storage of the original data. When storing, the system adds a predefined data tag to the data packet. The tag identifies the source and data type of the data to ensure that the subsequent modules can correctly distinguish the data type when processing; The status monitoring submodule first collects the data receiving rate, the rate after data format conversion, and the current number of data entries in the buffer storage unit from the data acquisition module; at the same time, through the interface communication with the downstream distributed data processing module, the real-time processing status of the downstream module, such as the data processing queue depth or the processing delay index, is collected. The status information is packaged in a unified format and passed to the condition judgment submodule; when the data transmitted by the status monitoring submodule indicates that a certain parameter exceeds the preset threshold, the condition judgment submodule compares it through the internal condition judgment logic. After judgment, if there is a matching update condition in the rule base, the condition judgment submodule generates the corresponding update request information and passes the judgment result to the configuration update submodule; the configuration update submodule constructs a new configuration parameter package, which specifically includes the working parameters of each module, such as the data buffer size, the data acquisition rate, etc., and performs version encoding on the parameter package. Through the system's built-in microservice interface, the parameter package is sent to the data acquisition module and other downstream modules connected to it in a message format to ensure that each module of the system receives a unified configuration update instruction.

[0024] See also Figure 1 and Figure 2 , an embodiment provided by the present invention: a big data analysis and display system with dynamic configuration, the system includes a distributed data processing and analysis module, relies on a distributed computing platform to realize data preprocessing, parsing and data mining, and internally integrates a deep reinforcement learning intelligent agent; The intelligent agent simultaneously collects the data transmitted by the data collection module and the configuration parameters issued by the dynamic configuration management module, generates parameters for data segmentation, task scheduling, and resource allocation based on the internal state prediction model, and feeds them back to the dynamic configuration management module; The distributed data processing and analysis module is internally equipped with a deep reinforcement learning intelligent agent, which includes a historical state collection unit, a real-time state update unit, and a policy generation unit; The historical state collection unit collects historical data and status information from the data collection module and the dynamic configuration management module; The real-time state update unit continuously receives the current processing status; The policy generation unit generates specific data segmentation, task scheduling, and resource allocation parameters based on the set neural network structure, and feeds the generated parameters back to the dynamic configuration management module for further adjustment of downstream processing; The distributed data processing and analysis module also has a data prediction linkage sub-module. The data prediction linkage sub-module uses a Transformer or LSTM network to implement time-series data feature prediction. The prediction results and the current data stream characteristics are transmitted to the dynamic configuration management module through a predefined interface to update the trigger conditions in the rule library and achieve early warning of the data stream state; Furthermore, the data collected by the historical state collection unit includes the timestamp, data type identifier, data volume, anomaly mark, etc. output by the data collection module. At the same time, the previously issued configuration parameters and status feedback are obtained from the dynamic configuration management module. The collected data is stored in the internal historical state database in a time series manner for subsequent neural network training and policy generation. The real-time state update unit subscribes to the status update events of the distributed processing platform, such as the processing progress, delay, queue depth, and resource utilization rate of each task. The received status data is transmitted to the policy generation unit inside the intelligent agent in a standardized JSON format through the internal bus. The real-time state data is matched with the historical data on the time axis to provide the current context information for the neural network. The policy generation unit uses a preset neural network structure, such as a convolutional neural network combined with a long short-term memory network, to predict the current state and generate specific operation parameters. The input end of the neural network model receives the combined data from the historical state collection unit and the real-time state update unit. During the training process of the model, a state-policy correspondence relationship is constructed using a supervised learning or reinforcement learning framework. The training objective is to output specific parameters such as data segmentation granularity, task scheduling priority, and resource allocation ratio. After the output parameters form a policy data packet, they are fed back to the dynamic configuration management module through a predefined microservice interface after internal encoding and version confirmation for the latter to update the downstream processing configuration; The data prediction linkage sub-module uses a Transformer network as a time-series data feature prediction model. The model training data comes from the time-series data and key status indicators recorded by the historical status collection unit. The preprocessing process includes data normalization, noise filtering, and time window segmentation, etc., to ensure that the input data meets the requirements of the neural network. After training, the model is saved as a standardized model file for online inference. The sliding window method is used to perform real-time prediction on the input data sequence, generating predicted values of key indicators within a future time series. The prediction results and the characteristics of the real-time data stream are encapsulated into data packets and transmitted to the dynamic configuration management module through predefined interfaces. The dynamic configuration management module updates the trigger conditions in its internal rule library based on the received prediction data, bringing possible future data states into the monitoring scope in advance.

[0025] Please refer to Figure 1 and Figure 2 , an embodiment provided by the present invention: a big data analysis and display system with dynamic configuration. The system includes a data display module, which is implemented using a front-end and back-end separation architecture. The front-end renders charts based on the React framework and uses WebSocket to receive data from the distributed data processing and analysis module and display parameters passed by the dynamic configuration management module in real time to achieve chart linkage and adaptive layout; The data display module includes a data parsing sub-module and a multi-view rendering sub-module; The data parsing sub-module parses the output data from the distributed data processing and analysis module using format standardization logic and packs the data according to the display parameters issued by the dynamic configuration management module; The multi-view rendering sub-module parses the parameter data according to the information linkage rules and renders various visual charts through SVG, and supports linkage calls and data synchronization between charts; Furthermore, when the front-end receives a data packet, the data parsing sub-module first performs data preprocessing. This sub-module contains format standardization logic, and its processing process is as follows: detect the structure of the "data part" in the data packet and compare it with the predefined format template; convert the original data into the internal standard format of the system according to the predefined rules, such as renaming each data item to a unified field name, converting the date string to a timestamp, and normalizing various measurement values; combine the parameters in the "configuration part", such as the specified chart data range, partitioning standard, data screening conditions, etc., to perform secondary packaging on the data and generate a structured data object required for rendering; After receiving the data packaging object transmitted by the data parsing sub-module, the multi-view rendering sub-module first performs information linkage rule parsing: pre-load the display parameters and linkage rules issued by the dynamic arrangement and management module. These rules include the linkage logic between charts and the specific attribute definitions of each chart; perform rule matching on the parsed data object to determine which chart each data field should be mapped to, and at the same time parse the drawing parameters of each chart. After the parsing is completed, the multi-view rendering sub-module uses SVG technology to generate visual charts according to the matched parameters, calls the SVG rendering library inside the React component to draw the corresponding charts according to the parsed data, calculates the display position, size, and spacing of each view according to the layout parameters defined in the configuration file to ensure adaptive layout at different resolutions, embeds event listeners in the React component, and when the user performs interactions such as clicking, zooming, or mouse hovering in any chart, the event information is passed to other chart components through the internal event dispatching mechanism to achieve linkage calls and data synchronization; for example, when the user selects a time interval in the line chart, other charts automatically refresh the data statistics information for that time period.

[0026] Please refer to Figure 1 and Figure 2 For an embodiment provided by the present invention: a big data analysis and display system with dynamic configuration, the status monitoring sub-module in the dynamic configuration management module periodically collects operation parameters from the data acquisition module, the distributed data processing and analysis module, and the data display module. The operation parameters include packet rate, delay, and load status, and are transmitted to the condition judgment sub-module in a unified format. The condition judgment sub-module has a rule library based on multi-layer rule judgment logic, and the rule library stores the critical state thresholds of each module and the corresponding configuration parameter sets for subsequent real-time calls by the configuration update sub-module. The configuration update sub-module in the dynamic configuration management module realizes data interaction with each downstream module through a microservice interface. The configuration update sub-module includes a configuration instruction encoding unit and a version control unit. The configuration instruction encoding unit standardizes and packages the update parameters, and sends the instructions to the distributed data processing and analysis module and the data display module through a message queue. At the same time, after receiving the feedback information, it forms a two-way communication with the status monitoring sub-module. Further, the condition judgment sub-module has a built-in multi-layer rule library, in which the critical state thresholds and corresponding configuration parameter sets of each module are preset. For example: rules for the data acquisition module, when the data packet rate exceeds the preset upper limit, such as 1300 packets / second, or the delay exceeds 50 milliseconds, the preset update parameters are "reduce the acquisition frequency", "increase the buffer", etc.; rules for the distributed data processing module, when the processing delay exceeds 70 milliseconds or the node load CPU > 80% reaches the critical state, the preset update parameters are "adjust the task scheduling strategy" or "dynamically increase or decrease computing resources", etc.; rules for the data display module, when the chart update delay or the linkage response delay exceeds the preset value, the preset update parameters are "optimize the data parsing rate" or "adjust the display refresh interval", etc.; after the condition judgment sub-module receives the data in the unified format transmitted by the status monitoring sub-module, it makes a comparison according to the preset multi-layer rule judgment logic. If any index in the data exceeds its preset threshold, the corresponding configuration parameter set is selected to generate an update request record. For example, in data processing, if the "processing delay" reaches 80 milliseconds and the CPU load is 82%, the condition judgment sub-module will determine that the update operation parameters are "reduce the data segmentation granularity", "adjust the scheduling queue priority", etc., and form a standardized update request data packet with this information and then transmit it to the configuration update sub-module; The configuration update sub-module is responsible for converting the update request generated by the condition judgment sub-module into a standardized configuration instruction and performing data interaction with the downstream module through the microservice interface. The specific process is as follows: Configuration instruction encoding unit, parameter standardization: encapsulate the parameter set passed in by the condition judgment sub-module to generate an instruction data packet in a unified format and attach the instruction version number; Message queue distribution, the configuration update sub-module issues the standardized instruction data packet to the specified downstream distributed data processing and analysis module and data display module through the predefined message queue to ensure that each module can receive the new configuration instruction in a timely manner; Version control unit, to ensure the consistency and traceability of the instruction, the version control unit records the version information and sending time of each instruction update, and at the same time compares it with the feedback information of each downstream module to confirm that the instruction has been correctly received and executed; Two-way feedback interaction, after the downstream module receives the configuration instruction, it performs the configuration update and feeds back the update execution status, such as "update successful" or "partial parameter update failed", through the same message queue or directly calling the microservice interface. After receiving the feedback information, the configuration update sub-module transmits the feedback data to the status monitoring sub-module to form a two-way data interaction loop for subsequent re-sampling and rule judgment of the system status, realizing closed-loop monitoring and dynamic configuration update.

[0027] Please refer to Figure 1 and Figure 2, an embodiment provided by the present invention: a big data analysis and display system with dynamic configuration. The communication unit between each module is connected by a microservices architecture. The communication unit includes a RESful API call interface, a gRPC communication interface, and a message queue interface based on RabbitMQ. A unified communication protocol parser is embedded in each module to achieve unified encoding, decoding, and verification of the data packet format, ensuring the interoperability and synchronization of the data stream and control instruction transmission between each module; The system adopts a containerized deployment structure. All modules are encapsulated in independent Docker containers and are orchestrated and managed through a Kubernetes cluster. The Kubernetes cluster is configured with an automatic scaling unit, a service discovery module, and a node health detection unit; The service discovery module is linked with the dynamic configuration management module, which is used to periodically detect the status of each container and feedback the container status data to the dynamic configuration management module to ensure the consistency of the internal logic scheduling of the system; There is a unified state synchronization control platform inside the system. It receives the status and log information from the data collection module, the dynamic configuration management module, the distributed data processing and analysis module, and the data display module through a dedicated data collection interface, performs real-time unified monitoring based on the standardized data aggregation mechanism of Prometheus, and feeds back the aggregation result to the dynamic configuration management module to achieve the logical linkage, verification, and closed-loop update of the configuration parameters and status information of the entire system; Furthermore, each module in the system, such as the data collection module, the dynamic configuration management module, the distributed data processing and analysis module, the data display module, etc., are respectively encapsulated in independent Docker containers to ensure the isolation of the running environment between modules and the unity of dependencies. Each container image contains a predefined communication protocol parser, built-in log collection, and health monitoring functions, enabling the container to provide a standardized service interface externally. Kubernetes automatically adjusts the number of container replicas of each module according to the container load, and ensures that resources can be dynamically scheduled in case of high concurrency or emergencies through the Horizontal Pod Autoscaler. The Kubernetes cluster embeds a service discovery mechanism such as CoreDNS, and each module calls each other through the service name. The service discovery module is linked with the dynamic configuration management module to periodically detect the running status of all containers, such as the online status, response time, and health check results, and feedback these container status data to the dynamic configuration management module. Kubernetes regularly checks the health status of each node to ensure the stability of the container deployment environment. Abnormal nodes will be isolated or rebuilt to maintain the consistency of the logical scheduling and configuration parameters of the entire system; There is a unified status synchronization control platform inside the system. Each module sends status information and log data to this platform through a dedicated interface. The status data includes the operating parameters, response latency, data packet rate, exception logs, etc. of each module. After the log data is converted into a unified format, it forms a standardized data packet, which is convenient for aggregation and subsequent analysis. The status synchronization control platform uses monitoring tools such as Prometheus and queries the status data of each module regularly in a standardized collection protocol such as the PromQL query language. The aggregated monitoring data is fed back to the dynamic configuration management module through a dedicated data interface, enabling it to verify the configuration parameters of each module based on the global status information and then trigger necessary configuration updates to form a complete closed-loop management.

[0028] Working principle: The system first collects raw data from multiple heterogeneous data sources through predefined interfaces, and uniformly converts formats such as JSON, XML, and CSV into a standard format via a data format conversion unit and temporarily caches it using a data buffer storage unit. The dynamic configuration management module is started based on an event-driven architecture. Its internal status monitoring sub-module regularly collects the real-time operating parameters of the data collection module and downstream modules and passes them to the condition judgment sub-module in a unified format, and then judges whether system configuration needs to be adjusted. The collected data enters the distributed data processing and analysis module, where data is preprocessed, parsed, and deeply mined based on big data computing platforms such as Spark and Flink. The deep reinforcement learning intelligent agent integrated inside the module generates specific parameters for data segmentation, task scheduling, and resource allocation based on a preset neural network model through a historical state collection unit and a real-time state update unit, and feeds the generated strategy back to the dynamic configuration management module. The data prediction linkage sub-module uses a Transformer or LSTM model to perform temporal prediction on the future data stream characteristics and passes it to the dynamic configuration management module in advance for updating preset rules. After the data processing is completed, the processing results are passed to the data display module through the distributed data processing module. The front end adopts a front-end and back-end separation architecture based on React and WebSocket real-time communication to achieve chart linkage, adaptive layout, and multi-view display. At the same time, the data display module feeds back the user interaction data and display status to the dynamic configuration management module, forming a closed loop with the operating parameters between modules, and realizing unified communication and status synchronization of the entire system through microservice interfaces, gRPC, and a message queue based on RabbitMQ to ensure continuous optimization of full-link parameters and dynamic configuration updates.

[0029] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in any respect, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Accordingly, all changes that fall within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. A big data analysis and display system with dynamic configuration, including a dynamic configuration management module and a data acquisition module, characterized in that: The data acquisition module uses a predefined interface to collect data from multiple heterogeneous data sources. The dynamic configuration management module is based on an event-driven architecture and has a status monitoring submodule, a condition judgment submodule and a configuration update submodule. The state monitoring submodule periodically collects real-time states from the data acquisition module and downstream modules, and transmits the collected data to the condition judgment submodule; The condition judgment submodule relies on the rule base to judge the need for updating the configuration, and sends the updated parameters in the form of a microservice interface through the configuration update submodule.

2. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The data acquisition module is internally provided with a data format conversion unit and a data buffer storage unit; The data format conversion unit can support automatic conversion between JSON, XML and CSV formats; The data buffer storage unit uses memory database and local cache technology to temporarily store raw data, and transmits it to the dynamic configuration management module after identifying the data type through predefined data tags.

3. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The system includes a distributed data processing and analysis module, which relies on a distributed computing platform to implement data preprocessing, parsing and data mining, and integrates a deep reinforcement learning intelligent agent internally; The intelligent agent simultaneously collects the data passed by the data collection module and the configuration parameters issued by the dynamic configuration management module, generates parameters for data segmentation, task scheduling and resource allocation based on the internal state prediction model, and feeds them back to the dynamic configuration management module.

4. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The system includes a data display module, which is implemented using a front-end and back-end separation architecture. The front-end renders charts based on the React framework, and uses WebSocket to receive data from the distributed data processing and analysis module and display parameters transmitted by the dynamic configuration management module in real time, thereby realizing chart linkage and adaptive layout.

5. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The status monitoring submodule in the dynamic configuration management module periodically collects operating parameters from the data acquisition module, the distributed data processing and analysis module, and the data display module. The operating parameters include packet rate, delay, and load status, and are transmitted to the condition judgment submodule in a unified format. The condition judgment submodule has a built-in rule library based on multi-layer rule judgment logic. The rule library stores the critical state thresholds of each module and the corresponding configuration parameter sets for real-time calling by the subsequent configuration update submodule.

6. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: The configuration update submodule in the dynamic configuration management module implements data interaction with each downstream module through a microservice interface, and the configuration update submodule includes a configuration instruction encoding unit and a version control unit; The configuration instruction encoding unit encapsulates the update parameters in a standardized manner and sends the instructions to the distributed data processing and analysis module and the data display module through the message queue. At the same time, after receiving feedback information, it forms a two-way communication with the status monitoring submodule.

7. The big data analysis and display system with dynamic configuration according to claim 3, characterized in that: The distributed data processing and analysis module is internally provided with a deep reinforcement learning intelligent agent, which includes a historical state collection unit, a real-time state update unit and a strategy generation unit; The historical status collection unit collects historical data and status information from the data collection module and the dynamic configuration management module; The real-time status update unit continuously receives the current processing status; The strategy generation unit generates specific data segmentation, task scheduling and resource allocation parameters based on the set neural network structure, and feeds the generated parameters back to the dynamic configuration management module for further adjustment of downstream processing.

8. The big data analysis and display system with dynamic configuration according to claim 3, characterized in that: The distributed data processing and analysis module is also provided with a data prediction linkage submodule, which uses a Transformer or LSTM network to realize the prediction of time series data features. The prediction results and the current data flow characteristics are transmitted to the dynamic configuration management module through a predefined interface to update the trigger conditions in the rule base and realize early warning of the data flow status.

9. The big data analysis and display system with dynamic configuration according to claim 4, characterized in that: The data display module includes a data parsing submodule and a multi-view rendering submodule; The data parsing submodule uses format standardization logic to parse the output data from the distributed data processing and analysis module, and packages the data according to the display parameters issued by the dynamic configuration management module; After parsing the parameter data according to the information linkage rules, the multi-view rendering submodule renders various visualization charts through SVG, and supports linkage calls and data synchronization between charts.

10. The big data analysis and display system with dynamic configuration according to claim 1, characterized in that: Each module is connected by a communication unit of microservice architecture, which includes RESful API call interface, gRPC communication interface and RabbitMQ-based message queue interface. Each module is embedded with a unified communication protocol parser to achieve unified encoding, decoding and verification of data packet formats, ensuring interoperability and synchronization of data flow and control instruction transmission between modules. The system adopts a containerized deployment structure. All modules are encapsulated in independent Docker containers and orchestrated and managed through the Kubernetes cluster. The Kubernetes cluster is configured with automatic expansion and contraction units, service discovery modules, and node health detection units. The service discovery module is linked with the dynamic configuration management module to periodically detect the status of each container and feed back the container status data to the dynamic configuration management module to ensure the coordination and consistency of the internal logical scheduling of the system; A unified state synchronization control platform is set up inside the system, which receives status and log information from the data acquisition module, dynamic configuration management module, distributed data processing and analysis module, and data display module through a dedicated data acquisition interface. It performs real-time unified monitoring based on Prometheus's standardized data aggregation mechanism, and feeds the aggregation results back to the dynamic configuration management module to achieve logical linkage, verification, and closed-loop update of the configuration parameters and status information of the entire system.

Images