Big data resource supervision system

Abstract

The invention discloses a big data resource supervision system which comprises an infrastructure supervision module, a data resource supervision module, a business record supervision module, an operator supervision module, a system program supervision module, a data efficiency supervision module, a data storage module and a visual interface module. The security, accuracy and integrity of the data are guaranteed, and the efficiency of data management is improved.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically a big data resource monitoring system. Background Technology

[0002] Traditional data resource supervision systems have the following problems: (1) Each module works independently, and the data is stored in a scattered manner, forming "data silos". It can only realize basic query and display functions, cannot play the value of data collaboration, and is difficult to support complex regulatory analysis needs.

[0003] (2) Most of them adopt a separate link supervision mode, with each module operating independently, lacking effective control over the overall system link.

[0004] (3) The Agent probe program directly uploads all the collected data to the server for calculation. This method not only increases the computing pressure on the server, but also leads to excessive network transmission burden. Especially in scenarios with high data collection frequency and large data volume, it is easy to cause network congestion and server performance degradation.

[0005] (4) Under the traditional regulatory model, all data rely on the server for analysis and processing. Even simple data anomalies require a lot of server computing resources, resulting in resource waste. Furthermore, analysis delays are likely to occur when the network is unstable.

[0006] (5) Databases are often used to store operation logs. However, high-privilege personnel may modify the database records, which makes it impossible to guarantee the authenticity and integrity of the operation data. In the event of a security incident or when tracing responsibility, it is difficult to provide reliable evidence. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention provides a big data resource supervision system that enables comprehensive supervision of data resources, ensuring data security, accuracy, and integrity, and improving the efficiency of data management.

[0008] The technical solution adopted by this invention to solve its technical problem is: A big data resource supervision system includes: an infrastructure supervision module, a data resource supervision module, a business registration supervision module, an operator supervision module, a system and procedure supervision module, a data performance supervision module, a data storage module, and a visualization interface module. The infrastructure monitoring module includes: a monitoring submodule, which collects infrastructure operation data according to a preset cycle through an Agent probe program and saves it in the data storage module; and an analysis submodule, which uses an LSTM model to determine the operating status of equipment, triggers an early warning mechanism if an anomaly is detected, and displays the result in a visualization interface module. The data resource supervision module includes: a data classification submodule, which uses a BERT pre-trained model to perform semantic analysis on data content and combines a regular expression rule base to automatically classify and label data resources, and encrypts and stores data marked as highly sensitive; a data quality detection submodule, which uses a sampling detection unit to detect data, and if the detection times out, it pauses the detection and generates a task blocking report to notify the operation and maintenance personnel, and normally stores the detection results in the audit table; and a data security monitoring submodule, which collects network traffic through a mirror port, parses it through Suricata IDS and stores it in Elasticsearch, and stores the detection results in the data storage module. The business filing and supervision module includes: a filing information entry submodule, which automatically loads the corresponding template according to the business type for the business party to fill in online and electronically signs the filing documents; a filing information review submodule, which sets up a three-level review unit to review the filing information; and a filing information tracking submodule, which connects with the business system, automatically collects business operation data on a regular basis, evaluates the consistency between the actual business execution and the filing content, generates a deviation analysis report, and updates the filing information status. The operator supervision module includes: an identity authentication submodule, an operation behavior recording submodule, and an access control submodule; The system and procedure supervision module includes: a system library management submodule, which stores and manages various versions of data supervision systems and procedure documents; a compliance check submodule, which checks whether business operation processes comply with regulations. If non-compliant, it automatically generates a list of compliance defects and a non-compliance report, and links them to the specific responsible department; and a training management submodule, which provides online training courses and assessment functions for operators. The assessment results are linked to the access control module, and those who fail the assessment are automatically restricted from high-risk operation permissions. The data performance monitoring module includes: a data value assessment submodule, which quantitatively assesses the potential economic, strategic, and social value of data resources; a data usage efficiency analysis submodule, which analyzes data usage efficiency through indicators such as access frequency and usage duration of statistical data; and a decision support submodule, which integrates knowledge graph technology to link data assessment results with industry best practice cases and generate decision recommendation documents based on the data value assessment and usage efficiency analysis results, providing decision recommendations for the optimal allocation and utilization of data resources.

[0009] Furthermore, the data collected by the monitoring submodule is compressed using the gRPC protocol and transmitted to the Kafka message queue of the central server's data storage module. The Flink real-time computing platform then consumes the queue data, performs format conversion, and stores it in the ClickHouse database. When the Agent probe program fails to connect to the central server N times consecutively, it automatically switches to offline storage mode, temporarily storing the data in a local SQLite database, and uploads it in batches after the network is restored.

[0010] Furthermore, the LSTM model of the analysis submodule obtains infrastructure operation data from the data storage module and is automatically trained once a day. During the training process of the LSTM model, an Early Stopping strategy is adopted. When the error on the validation set does not decrease for 5 consecutive times, the training is stopped and the optimal model is saved. The prediction model is then updated based on the latest 7 days of data.

[0011] Furthermore, the analysis submodule includes a device health report generation unit, which automatically generates a device health report if the operating device malfunctions.

[0012] Furthermore, the data quality inspection submodule adopts the DQM framework and uses metadata lineage analysis technology to track the entire lifecycle of data from generation to use. For structured data, it sets rules such as field non-empty checks and value range verification, while for unstructured data, it uses similarity comparison algorithms to detect duplicate content.

[0013] Furthermore, the data security monitoring submodule also includes an anomaly detection unit. When the anomaly detection unit detects frequent access to sensitive data, it immediately triggers the circuit breaker mechanism to block the communication link. When the anomaly detection unit detects unauthorized access attempts, it immediately blocks the access and records the logs, while displaying the relevant security events on the visualization interface.

[0014] Furthermore, the operation behavior recording submodule is built on Apache Kafka to create a sand queue, which captures and records the operation logs of operators on data resources in real time, and realizes centralized storage, retrieval and visualization analysis of logs through ELK technology; Furthermore, the permission management submodule, based on the RBAC model, dynamically allocates or adjusts the scope of data access and operation permissions in real time according to the operator's position, responsibilities, and risk value. If an operator attempts to perform an operation beyond their permissions, the system automatically rejects it and records the log. Alternatively, if the system detects that an operator is accessing sensitive data abnormally frequently, it automatically downgrades their permission level.

[0015] Furthermore, the visualization interface module includes an interactive unit, where supervisors can issue operation commands and query data for each module; and it uses WebSocket technology to refresh data in real time. When new warning information is generated, the interface automatically pops up a notification box and plays a prompt sound.

[0016] The beneficial effects of this invention are: 1) The beneficial effects of cross-module data transfer protocols This invention employs a unified cross-module data flow protocol, using Apache Kafka as the message bus, and defines a standardized data flow format. This protocol forces each module to adhere to a unified standard for data exchange, enabling previously isolated data to flow orderly between modules and building an interconnected data network. For example, after the business registration and supervision module completes the review of registration information, relevant data can be seamlessly transferred to the data resource supervision module according to the protocol for data compliance checks, achieving deep data correlation applications. Simultaneously, the standardized protocol incorporates data traceability identifiers and system observation fields, supporting full lifecycle tracing of data from generation to use, facilitating rapid identification of data anomaly sources, improving supervision efficiency and accuracy, and truly realizing collaborative data sharing and value mining.

[0017] 2) The beneficial effects of the end-to-end regulatory system This invention introduces OpenTelemetry to construct a distributed system end-to-end monitoring framework, and uses Jaeger to visualize the end-to-end call relationships. When a system failure occurs, the visualization interface can quickly locate the specific faulty module and its upstream and downstream dependencies, achieving millisecond-level fault location. For example, when the data resource monitoring module experiences data processing delays, end-to-end monitoring can quickly determine that the cause is an abnormal data transmission from its upstream infrastructure monitoring module, avoiding blind troubleshooting and significantly shortening fault handling time. Simultaneously, end-to-end monitoring can monitor the data interaction and operational status between modules in real time, proactively identifying potential performance bottlenecks and security vulnerabilities, enhancing system stability and reliability, and ensuring the continuous and efficient operation of the entire monitoring system.

[0018] 3) The beneficial effects of edge data preprocessing This invention adds data filtering and aggregation functions to the Agent probe program of the infrastructure monitoring module in edge computing scenarios, uploading only key indicator data. For example, it calculates the average of data such as CPU utilization and memory usage every 10 minutes, and only uploads the aggregated data to the cloud. This improvement effectively reduces data transmission volume, lowers network bandwidth requirements, alleviates server computing pressure, and improves data processing efficiency. Simultaneously, edge preprocessing can quickly complete preliminary data processing locally, avoiding data processing delays caused by network latency and ensuring the timeliness and effectiveness of monitoring data.

[0019] 4) Beneficial effects of edge-server collaborative supervision This invention constructs an edge-server collaborative monitoring mechanism. When abnormal data (such as a sudden rise in device temperature) is detected at the edge, the edge-side analysis module first performs a preliminary diagnosis. If the problem can be handled, the corresponding operation is executed directly. If it cannot be handled, detailed data and diagnostic results are uploaded to the server. The server's powerful computing resources are then used for in-depth analysis, and the processing solution is distributed to the edge for execution. For example, for a slight increase in device temperature, the edge can directly initiate cooling measures; while for complex temperature anomalies, the server provides a more precise processing solution after in-depth analysis. This collaborative mode achieves reasonable allocation of computing resources, reduces ineffective server computation, improves resource utilization, and ensures that abnormal data is processed in a timely and effective manner, thereby improving the overall system performance and response speed.

[0020] 5) The beneficial effects of signature-based evidence storage of operational actions This invention employs an operator monitoring module to electronically sign and store all high-risk operations (such as deleting sensitive data and modifying core configurations), using blockchain and other encryption technologies to ensure the immutability of these operation records. Each high-risk operation generates a unique electronic signature, which is bound to the operation data and stored in a secure evidence storage system. Even personnel with high privileges cannot modify the stored data, ensuring the authenticity and integrity of the operation data. In the event of a data security incident or when accountability needs to be traced, these immutable operation records serve as conclusive evidence, accurately tracing the source of the operation, clarifying responsibility, effectively improving the system's security and credibility, and providing a solid guarantee for data security. Attached Figure Description

[0021] Figure 1 This is a system architecture diagram of the present invention; Figure 2 Flowchart for the infrastructure regulatory module; Figure 3 A flowchart for the data classification submodule; Figure 4 This is a flowchart of the data quality inspection submodule. Figure 5 The flowchart for the detection process of the data security monitoring submodule; Figure 6 A flowchart for the business filing and supervision module; Figure 7 A flowchart for the operator supervision module; Figure 8 Flowchart for the system and procedure supervision module; Figure 9 Data performance monitoring module flowchart; Figure 10 This is a flowchart of the data analysis and processing of the present invention; Detailed Implementation This invention includes an infrastructure supervision module, a data resource supervision module, a business registration supervision module, an operator supervision module, a system and procedure supervision module, a data performance supervision module, a data storage module, and a visualization interface module.

[0022] The following is a detailed description of each module in conjunction with the accompanying drawings. It should be noted that, in order to make the solution of the present invention easier to understand, the following description adopts a module-based flowchart approach, and the process design of each module is implemented by the module device.

[0023] 1) Infrastructure Supervision Module: This module corresponds to the attached... Figure 1 Module A in the module mainly consists of a monitoring submodule and an analysis submodule. The monitoring submodule uses a lightweight agent program as a probe, and implements data collection and visualization based on the Prometheus + Grafana technology stack. The agent program is deployed on nodes of infrastructure such as servers, storage devices, and network facilities, and periodically (by default every 5 minutes) pushes device operation data, such as CPU utilization, CPU core temperature, memory usage, network bandwidth, network packet loss rate, and disk I / O throughput, to the central server via HTTP coordination.

[0024] The analysis submodule uses a preset LSTM (Long Short-Term Memory) algorithm model to perform time series prediction of device performance trends on the collected historical data. If the prediction results show that the CPU utilization of a certain server exceeds the threshold of 85% continuously within the next hour, an early warning mechanism will be triggered immediately. At the same time, the device status will be highlighted in red in the visualization interface module in the attached figure, and a needle / email alarm will be sent to the operation and maintenance personnel via Webhook.

[0025] 2) Data Resource Supervision: Module B in the attached diagram, comprising a data classification submodule, a data quality inspection submodule, and a data security monitoring submodule. The data classification submodule is based on Natural Language Processing (NLP) technology, employing a BERT pre-trained model to perform semantic analysis on the data content, and combining it with a regular expression rule base to automatically classify and label data resources. For example, when text data containing ID card numbers or mobile phone numbers is detected, it will be automatically marked as "highly sensitive data" containing user personal information and encrypted using the national cryptographic algorithm SM4. The data quality detection submodule adopts the DQM (Data Quality Management) framework and tracks the entire lifecycle of data from generation to use through metadata lineage analysis technology. For structured data, it sets rules such as field non-empty checks and value range checks, while for unstructured data, it uses similarity comparison algorithms to detect duplicate content. The data security monitoring submodule is based on a zero-trust architecture and deploys a micro-segmentation strategy. It analyzes the access, transmission, and storage processes of data at the network layer, transport layer, and application layer in real time through traffic mirroring technology. When abnormal data transmission behavior is detected, such as high-frequency access to sensitive data in a short period of time, the circuit breaker mechanism is immediately triggered to block the communication link. Or when illegal access behavior is detected, such as an unauthorized user attempting to access sensitive data, access is immediately blocked and logs are recorded. At the same time, the relevant security events are displayed on the visualization interface.

[0026] 3) Business Filing Supervision: Corresponding to Module C in the attached diagram, there are three sub-modules: Filing Information Input, Filing Information Review, and Filing Information Tracking. The Filing Information Input sub-module uses Vue.js + Element UI to build standardized front-end filing forms, supporting dynamic form generation. It can automatically load corresponding templates based on business type, allowing business parties to fill in relevant business information online. It also introduces electronic signature technology (based on a PKI system) to achieve online signing of filing documents, ensuring the documents are tamper-proof. The Filing Information Review sub-module has a three-level review process: initial review is completed by an intelligent review engine (based on a rule engine matching keywords and logical conditions); secondary review is conducted manually by business supervisors through the review interface, manually reviewing the filing information according to review rules, such as checking whether the business process is compliant; and final review is confirmed by the compliance department. The Filing Information Tracking sub-module connects to the business system through an API interface, automatically collecting business operation data quarterly, using comparative analysis algorithms to evaluate the consistency between actual business execution and the filing content, generating a deviation analysis report, and updating the filing information status.

[0027] 4) Operator Supervision: Module D in the attached diagram consists of an identity authentication submodule, an operation behavior recording submodule, and an access control submodule. The identity authentication submodule adopts the OAuth 2.0 + JWT (JSON Web Token) authentication mechanism, supporting biometric recognition (fingerprint / face) and dynamic token multi-factor authentication to ensure the authenticity of the operator's identity. The operation behavior recording submodule is built on Apache Kafka to build a sand queue, capturing and recording the operator's operation log behavior on data resources in real time, including SQL statements, API call parameters, operation time, operation content, etc., and realizing centralized storage, retrieval, and visualization analysis of logs through ELK (Elasticsearch-Logstash-Kibana) technology. The access control submodule is based on the RBAC (Role-Based Access Control) model and introduces the ABAC (Attribute-Based Access Control) dynamic extension mechanism. It can dynamically allocate or adjust the scope of data access and operation permissions in real time according to the operator's position and responsibilities and risk value (derived from the analysis of historical operation records). If the operator attempts to perform an operation beyond their permission, the system will automatically reject it and record the log, or if it detects that the operator accesses sensitive data abnormally frequently, it will automatically downgrade their permission level.

[0028] 5) System and Procedure Supervision: Module E includes a system library management submodule, a compliance inspection submodule, and a training management submodule. The system library management submodule uses the Git version control system to store and manage various versions of data supervision system and procedure documents, supporting multi-user collaborative editing and change traceability. The compliance inspection submodule uses process mining technology to model business operation logs, comparing and analyzing business operation processes with system procedures to check for compliance. If data transmission processes are found to be not executed according to encryption requirements, a compliance defect list and non-compliance report are automatically generated and linked to the specific responsible department. The training management submodule develops online courses based on the SCORM standard, integrates facial recognition technology for identity verification, and uses an adaptive learning algorithm to dynamically adjust the course difficulty based on student responses, providing online training courses and assessment functions for operators. Assessment results are linked to the access control module; those who fail the assessment are automatically restricted from high-risk operations to ensure their familiarity with relevant systems and procedures.

[0029] 6) Data Performance Supervision: Module F in the attached diagram, consisting of a data value assessment submodule, a data usage efficiency analysis submodule, and a decision support submodule. The data value assessment submodule constructs a multi-dimensional assessment indicator system, combines the Analytic Hierarchy Process (AHP) and entropy weight method to determine indicator weights, and uses a value assessment model to quantitatively evaluate the potential economic, strategic, and social value of data resources. The data usage efficiency analysis submodule collects data usage behavior through tracking technology, optimizes data access links using funnel analysis and path analysis, and analyzes data usage efficiency by collecting indicators such as access frequency and usage duration. The decision support submodule integrates knowledge graph technology, links data assessment results with industry best practice cases, and automatically generates decision recommendation documents based on the data value assessment and usage efficiency analysis results using natural language generation (NLG) technology. This provides decision recommendations for the optimized allocation and utilization of data resources, supports export in Word / PDF format, and displays the analysis report and decision recommendations in a visual interface.

[0030] 7) Visualization Interface Module: The common module shown in the attached diagram uses ECharts 5.0 for data visualization, supporting 12 visualization types including 3D charts and heatmaps. It displays monitoring data, analysis results, and early warning information from various regulatory modules. A data dashboard template library is introduced, allowing users to customize the layout of regulatory dashboards through drag-and-drop, presenting data in intuitive formats such as charts and reports. This facilitates quick understanding of data resource supervision by regulatory personnel. Interactive functions are also provided, allowing regulatory personnel to issue operation commands and query data through the interface. WebSocket technology is integrated to achieve real-time data updates. When new early warning information is generated, a notification box automatically pops up on the interface, and an alert sound plays. 8) Data Storage Module: As the core of the entire system's data storage, corresponding to the basic module in the attached diagram, it adopts the Ceph distributed storage system, constructing a three-replica redundant storage strategy to ensure data reliability. It is used to store various types of collected data, analysis results, log information, etc. Distributed storage technology is used to ensure data security and reliability. ClickHouse columnar database is introduced for real-time data analysis, combined with a Redis caching cluster to improve the performance of high-frequency data access and provide efficient data retrieval and access interfaces for other functional modules to call. Apache Iceberg is used for data lake management, supporting data version rollback and change auditing.

[0031] The present invention will be further described below through a process method description. It should be noted that the technical solutions described in this part belong to the above modules and together with the above modules form an overall solution.

[0032] 1) Data Acquisition Phase (1) The monitoring submodule of the infrastructure supervision module collects infrastructure operation data according to a preset cycle through the Agent probe program. The data is compressed by the gRPC protocol and transmitted to the Kafka message queue of the central server data storage module. The Flink real-time computing platform consumes the queue data, performs format conversion, and stores it in the ClickHouse database. When the Agent probe program and the central server time out (three consecutive failed requests), it automatically switches to offline storage mode and temporarily stores the data in the local SQLite database. The data will be uploaded in batches after the network is restored. (2) The data classification submodule of the data resource supervision module triggers the data resource classification and labeling task when the data is written to the storage system, and calls the BERT model for real-time classification; the data quality detection submodule adopts a sampling detection strategy (by default, 10% of the data is sampled every hour). If the detection task takes more than 15 minutes (default threshold), the detection is automatically paused and a task blocking report is generated to notify the operation and maintenance personnel to investigate the performance bottleneck. Under normal circumstances, the detection results are stored in the audit table of the data storage model through the RESTful API; the data security monitoring submodule collects network traffic through the mirror port, parses it through SuricataIDS and stores it in Elasticsearch, and stores the detection results in the data storage module. (3) The filing information entry submodule of the business filing supervision module will convert the signed filing information file submitted by the business into PDF format and store it in Ceph object data storage; the filing information tracking submodule will collect the running data of the filed business periodically through a scheduled task (calling the business system API on the 1st of each month), compare the collected data with the historical filing information by Diff, generate change records, and update the storage. (4) The identity authentication submodule of the operator supervision module generates a JWT token when the operator logs in, stores it in the Redis cache, and performs identity verification; the operation behavior recording submodule filters and cleans the operation log behavior data recorded in real time in Kafka through Logstash and writes it into Elasticsearch for storage; the permission management submodule synchronizes the organizational structure data and updates the operator permission data on a regular basis (every day at midnight) according to the configuration information, updates the permission configuration table and stores it in the data storage module. (5) The system library management submodule of the system and procedure supervision module stores and manages the system and procedure file records changed in the Git repository and synchronizes them to the data storage module; the compliance inspection submodule stores the inspection results in JSON format through the business process model generated by the process mining algorithm; the training management submodule stores reward learning records, training records and assessment results to the data storage module. (6) The data value assessment submodule of the data performance supervision module extracts metadata from each business system and performs assessment calculations in combination with external market data (obtained through web crawlers); the data usage efficiency analysis submodule collects user behavior data through the front-end embedding SDK and transmits it to the data storage module via Flume. 2) Data Analysis and Processing Stage (1) The LSTM model of the analysis submodule of the infrastructure supervision module obtains infrastructure operation data from the data storage module and is automatically trained once a day. During the training of the LSTM model, the Early Stopping strategy is adopted. When the error of the validation set does not decrease for 5 consecutive times, the training is stopped and the optimal model is saved to avoid overfitting. Under normal circumstances, the prediction model is updated based on the latest 7 days of data and the algorithm model is used for analysis to determine whether the equipment operation status is normal. If abnormal, the early warning mechanism is triggered and the equipment health report is automatically generated, including the analysis results such as failure probability and impact range.

[0033] (2) The data resource supervision module generates a data asset list weekly, which includes information such as data classification and distribution, quality score, and security risk level. Based on the stored data, it conducts a comprehensive analysis of the data classification, quality, and security status, and generates a data resource supervision report. For high-risk data, it automatically triggers a security hardening process (such as re-encryption and access permission adjustment).

[0034] (3) The intelligent review engine of the filing information review sub-module of the business filing supervision module is implemented by the rule engine (Drools), which supports dynamic rule updates, retrieves filing information from the data storage module, and conducts manual review; the deviation analysis report generated by the filing information tracking sub-module extracts key issues through NLP technology, automatically assigns them to the corresponding rectification responsible persons, analyzes the collected data, and evaluates the business filing status.

[0035] (4) The operator supervision module identifies violations based on the anomaly detection algorithm (Isolation Forest). When a suspicious behavior is detected, a risk assessment report is automatically generated. The operator's operation behavior data is analyzed to determine whether there is a violation. The permission management submodule dynamically adjusts the operator's permissions based on the operation behavior analysis results. After adjusting the permissions according to the risk value, a permission change notification email is sent to the operator.

[0036] (5) The compliance inspection submodule of the system and procedure supervision module performs graph comparison and matching analysis between business process operation data and system rules and procedures, calculates the compliance deviation using the shortest path algorithm, and generates a compliance report; the training management submodule analyzes training and assessment data through learning analytics (LDA) technology, evaluates trainees’ knowledge mastery and training effectiveness, and automatically assigns make-up exam tasks to trainees who do not meet the standards. (6) The decision support submodule of the data performance monitoring module updates the data knowledge graph daily, obtains data value assessment and usage efficiency analysis results from the data storage module, and automatically generates data optimization suggestions when the data value score changes by more than 20%. When generating data optimization suggestions, reinforcement learning algorithms are introduced to adjust the suggestion strategy based on historical optimization effect feedback. For example, if the usage efficiency is significantly improved after the execution of a certain type of data migration strategy, the recommendation weight of the strategy is increased. Combined with the data usage efficiency analysis results, the data storage strategy is adjusted through automated scripts (such as the business requirement of tiered storage of cold and hot data) to generate decision suggestions. 3) Results Presentation and Feedback Phase (1) Each regulatory module subscribes to the analysis results of each regulatory module through WebSocket and transmits the analysis results to the visualization interface module to display them to the regulatory personnel in the form of charts, reports and other forms. The charts are automatically refreshed when there is new data. The system supports data drill-down function. Clicking on the chart elements can view the detailed data list. The system supports multi-dimensional filtering function. The regulatory personnel can filter data by combining conditions such as time range, data type, and risk level. The system refreshes the charts in real time to display the filtering results.

[0037] (2) Supervisory personnel issue operation instructions through the visual interface based on the information displayed on the interface, such as restarting the server, banning user accounts, handling abnormal situations, and approving business filings. After authorization verification, the instructions are sent to the target functional module through the message queue. The operation instructions are accompanied by a version number to ensure idempotency in a distributed environment. In addition to role-based authorization verification, an operation frequency limit is added. If the same user initiates more than 5 high-risk operation instructions within 1 minute, secondary identity verification (such as SMS verification code) is required.

[0038] (3) Each functional module of the system performs corresponding operations and records operation logs according to the operation instructions of the supervisors. It adjusts and controls the relevant functional modules. For example, the infrastructure supervision module restarts the equipment according to the instructions, and the operator supervision module modifies the operator's permissions according to the instructions. After the operation is completed, the execution result is returned to the visualization interface for display. At the same time, the operation record is stored in the data storage module to trigger the update and collection process of relevant data, complete a complete supervision workflow, and form a supervision closed loop.

[0039] like Figure 2 As shown, the infrastructure oversight module process is as follows: 1. Data acquisition unit collects data: 1) Probe deployment: Agents are deployed on infrastructure nodes (servers / storage / networks).

[0040] 2) Data flow: Data acquisition (5-minute cycle) -> Normal network connectivity -> gRPC compression -> Kafka message queuing -> Flink real-time processing -> ClickHouse storage. It should be noted that the symbol "->" above only indicates the direction of information flow and has no other special meaning; the same applies below.

[0041] 3) Offline disaster recovery: If the network connection fails (3 consecutive connection failures) -> local SQLite temporary storage -> batch upload after network recovery.

[0042] 2. Analysis and Processing Unit: 1) Model training unit: LSTM daily training -> Early Stopping (error does not decrease after 5 consecutive validations) -> Save the best model.

[0043] 2) Prediction logic unit: Based on 7 days of data -> LSTM prediction -> determine if the CPU utilization rate is greater than the threshold of 85% in the next 1 hour.

[0044] 3) Early warning linkage unit: Triple response mechanism: visual interface highlighting, DingTalk / email notification, and generation of health report.

[0045] like Figure 3 As shown, the classification process of the data classification submodule is as follows: data writing -> triggering the classification task unit -> calling the BERT model -> semantic analysis + regular expression matching -> sensitive data detection unit performs detection -> yes -> marking and encryption -> SM4 storage. If it is not sensitive data, the data is stored in a standard manner.

[0046] Figure 4 The detection process for the data quality inspection submodule is as follows: the sampling unit performs hourly sampling inspections -> determines whether a timeout has occurred -> if yes, generates a blockage report -> notifies operations and maintenance; if no, performs DQM inspection -> lineage analysis -> results storage.

[0047] Figure 5 The detection process for the data security monitoring submodule is as follows: traffic mirroring -> Suricata parsing -> storage in Elasticsearch -> persistent storage. If Suricata parsing detects abnormal behavior, it determines the type of abnormal behavior. If it is high-frequency access to sensitive data, it triggers a circuit breaker mechanism to block the communication link; if it is an unauthorized access attempt, it immediately blocks the access. The abnormal behavior is recorded in the security log and displayed in the visualization interface module.

[0048] Figure 6 The process for the business filing and supervision module: 1. Data Acquisition Phase 1) The filing information entry submodule first constructs a dynamic form -> the business party fills in the dynamic form -> electronic signature -> the PKI system ensures that the electronic signature is tamper-proof -> convert to PDF -> store in Ceph object storage.

[0049] 2) The filing information tracking submodule starts a scheduled task on the 1st of each month -> collects API data -> performs Diff comparison -> generates change records -> updates storage.

[0050] 2. Data Analysis and Processing Stage 1) The filing information review submodule retrieves filing information -> initial review by the engine -> secondary review after initial review -> final review after secondary review. Failed reviews are returned for modification. A three-level review process is adopted: intelligent initial review, secondary review by business supervisor, and final review by compliance department. The Drools rule engine supports dynamic rule updates.

[0051] 2) The filing information tracking and analysis submodule automatically generates deviation analysis reports -> automatically extracts key issues using NLP technology -> automatically assigns them to the responsible persons for rectification -> assesses the business filing status.

[0052] Figure 7 The process for the operator supervision module: 1. Data Acquisition Phase 1) Identity authentication submodule: When authenticating user login, biometrics or dynamic tokens are used. After the JWT token is generated, it is stored in the Redis cache.

[0053] 2) Operation behavior recording submodule: Operation behavior capture unit captures behavior -> Kafka queue -> Logstash filtering and cleaning -> Elasticsearch storage -> Kibana visualization.

[0054] 3) Permission management submodule: Synchronize organizational structure data every day at midnight -> update permission configuration -> store to data storage module.

[0055] 2. Data Analysis and Processing Stage 1) Abnormal behavior detection unit: Operation log -> Lsolation_Forest algorithm -> Suspicious behavior? -> Yes -> Generate risk assessment report. Otherwise, record normally.

[0056] 2) Dynamic permission adjustment unit: Risk report -> RBAC+ABAC model -> Dynamic permission adjustment -> Email notification.

[0057] 3) Real-time security response unit: operation attempt -> unauthorized or high-frequency access -> automatic rejection -> log recording.

[0058] 4) Automatic permission degradation unit: Degradation is triggered from security log -> permission level is reduced -> operator is notified.

[0059] Figure 8 For the system and procedure supervision module process: This module's workflow consists of the training management sub-module workflow: student login -> facial recognition verification -> SCORM standard courses -> adaptive learning -> assessment and testing -> storing learning records -> storing training records -> storing assessment results -> synchronizing to the data storage module.

[0060] Compliance inspection submodule process: Business operation log -> Process mining technology -> Business process modeling -> Inspection results in JSON format -> Stored in data storage module.

[0061] The workflow of the policy repository management submodule is as follows: multi-person collaborative editing -> Git version control -> change tracking -> policy program file storage -> synchronization to the data storage module.

[0062] Compliance check and analysis process: graph comparison and matching analysis -> shortest path algorithm -> calculate compliance deviation -> comply with regulations? -> yes -> generate defect list -> generate non-compliance report -> associate with responsible department, if non-compliant -> compliance confirmation.

[0063] Training management analysis process: Learning Analytics (LDA) -> Evaluate training effectiveness -> Evaluate knowledge mastery. If the training effectiveness meets the standards, the assessment is passed; if it does not meet the standards, a re-examination task is analyzed; if the final assessment is not passed, access is restricted through the access management linkage unit.

[0064] Access control linkage unit: Failure to pass assessment -> Restrict high-risk operation permissions; Pass assessment -> Grant operation permissions; Analyze re-examination tasks -> Maintain permission restrictions.

[0065] Real-time compliance check process: Real-time compliance check -> Violation detection -> Defect list generation -> Automatic alert.

[0066] Figure 9 Data performance monitoring module process: Value assessment data collection submodule process: Obtain business system metadata and external market data -> Data value assessment.

[0067] The process of using the efficiency collection submodule is as follows: User behavior data -> Front-end event tracking SDK -> Flume transmission -> Storage to data storage module.

[0068] Value assessment and analysis sub-module process: Constructing an assessment indicator system -> Analytic Hierarchy Process (AHP) or Entropy Weight Method -> Determining indicator weights -> Quantifying value assessment -> Economic value / Strategic value / Social value.

[0069] The efficiency analysis unit workflow is as follows: funnel analysis / path analysis -> optimize access links -> access frequency statistics / usage duration analysis.

[0070] Decision Support Unit Process: Knowledge Graph Integration -> Association with Industry Best Practices -> Value Score Change -> Change greater than 20%, proceed to Value Change Response Unit -> Yes, generate optimization suggestions -> Reinforcement Learning Algorithm -> Historical Effect Feedback -> Adjust Suggestion Strategy -> NLG Technology -> Decision Suggestion Document -> Document Export / Visualization Interface Display; If the value score remains unchanged, maintain the current strategy.

[0071] The process of using the efficiency analysis unit involves optimizing the access link -> automating scripts for tiered storage of hot and cold data -> generating storage optimization solutions -> NLG technology -> decision recommendation documents -> document export / visualization interface display.

[0072] Value change response unit process: Trigger optimization process -> Reinforcement learning adjustment -> Increase the weight of effective strategies.

[0073] Figure 10 The data analysis and processing flow of this invention is as follows: 1. Weekly asset list -> Classification and distribution, quality score, risk rating -> Comprehensive report. When the risk rating is high risk, it will be re-encrypted and permissions will be adjusted.

[0074] 2. Traffic analysis -> Anomaly detection, high-frequency access -> Circuit breaker -> Unauthorized access -> Block -> Log recording -> Interface display.

[0075] In summary, the technical solution of the present invention has been described in detail. The parts not described in detail are all prior art and therefore have not been described in detail.

[0076] 1) Cross-module data flow protocol: The invention and improvement of this data resource supervision system lies in the six key data supervision modules: infrastructure supervision, data resource supervision, business filing supervision, operator supervision, system and procedure supervision, and data efficiency supervision. It adopts a unified cross-module data flow protocol, using Apache Kafka as the message bus. This standardized data flow protocol addresses the problem of existing supervision systems where each module operates independently, generating supervision data and storing it in different tables. These tables are independent, allowing only querying and display; the data remains isolated, like silos. Our cross-module data flow protocol ensures that all data exchanges between modules adhere to this protocol, guaranteeing data traceability and system observability. It's important to note that this cross-module data flow protocol is not merely a message format, but a comprehensive specification covering transmission, structure, semantics, and governance. This protocol involves the transport layer, structural layer, semantic layer, reliability layer, and governance layer. The transport layer specifies naming conventions, partitioning strategies, and consumer group strategies; the structure layer defines the specific appearance of each message, i.e., the message body structure; the semantic layer defines the meaning and behavior of messages; the reliability layer designs producer acknowledgment and consumer commit to ensure reliable configuration; and the governance layer designs access control and monitoring. Data flow and interaction between all modules must adhere to this protocol.

[0077] 2) End-to-End Monitoring System: The key improvement in this data resource monitoring system lies in its six data monitoring key modules. It introduces OpenTelemetry to achieve end-to-end monitoring of the distributed system. By visualizing the end-to-end call relationships using Jaeger, it enables millisecond-level fault location. This addresses the problem of existing monitoring systems operating on a separate model, which fails to effectively monitor the entire system and is prone to individual module key failures, leading to inadequate overall system monitoring. OpenTelemetry is an open-source observability framework that provides standardized SDKs and APIs for collecting, transforming, and sending application metrics, link, and log data. It is integrated throughout the data transmission of each sub-module. The above-mentioned end-to-end monitoring works as follows: First, starting from the data collection point, data points are embedded, which involves creating a Span element to record the start and end times. This Span element has an ID called SpanID. During context propagation of data resources, the globally unique identifier TraceID and the previous parent SpanID are packaged into a Context object and passed down through the carrier. During data processing, the Span data generated by the data points is first stored in memory, and then processed, such as adding or removing new resource tags. After processing, the Span data can be sent to the backend for storage or indexing, or it can be visualized, queried, and analyzed using Jaeger.

[0078] 3) Edge Data Preprocessing: The improvement of this data resource supervision invention lies in adding data filtering and aggregation functions to the Agent probe program of the infrastructure supervision module in edge computing scenarios. Only key indicator data is uploaded, reducing network transmission pressure. Market-based supervision systems upload all data collected by the Agent probe program to the server for calculation, increasing server and network transmission pressure. This invention, however, performs edge data preprocessing, solving this problem by averaging the collected data and uploading only the aggregated data to the server.

[0079] 4) Edge-Server Collaborative Monitoring: The improvement in this data resource monitoring invention is that when abnormal data is detected at the edge (such as a sudden increase in device temperature), the edge-side analysis module will first perform a preliminary diagnosis. If it cannot handle the situation, detailed data and diagnostic results will be uploaded to the server for in-depth analysis using powerful server computing resources, and the processing solution will be distributed to the edge for execution. This can solve the problem of wasted server computing resources.

[0080] 5) Operational Behavior Signature and Evidence Storage: The improvement in this data resource supervision invention lies in the operator supervision module's electronic signature and evidence storage for all high-risk operations (such as deleting sensitive data or modifying core configurations). This involves the system matching the risk database to confirm the type and freezing permissions, the operator verifying the detailed operation, supplementing authorization evidence, and selecting compliant methods such as hardware encryption or biometrics for signature, integrating the above operation record information to generate an evidence storage package. This ensures the immutability of operation records. General resource supervision typically uses databases for log storage, where high-privilege personnel can modify database records. This addresses the issues of authenticity, integrity, and tamper-proofing of operational data.

[0081] After the operation is completed, the integrated information generates a certificate storage package, whose hash value is entered into the blockchain for tamper-proof protection and is simultaneously backed up and archived offline. The authorized person can query the certificate storage after signing it. During auditing, the blockchain hash, CA signature, and data integrity are verified. Anomalies trigger alarms and trace the responsibility.

Claims

1. A big data resource monitoring system, characterized in that, It includes: The system includes modules for infrastructure supervision, data resource supervision, business registration supervision, operator supervision, system and procedure supervision, data efficiency supervision, data storage, and a visualization interface. The infrastructure monitoring module includes: a monitoring submodule, which collects infrastructure operation data according to a preset cycle through an Agent probe program and saves it in the data storage module; and an analysis submodule, which uses an LSTM model to determine the operating status of equipment, triggers an early warning mechanism if an anomaly is detected, and displays the result in a visualization interface module. The data resource supervision module includes: a data classification submodule, which uses a BERT pre-trained model to perform semantic analysis on data content and combines a regular expression rule base to automatically classify and label data resources, and encrypts and stores data marked as highly sensitive; a data quality detection submodule, which uses a sampling detection unit to detect data, and if the detection times out, it pauses the detection and generates a task blocking report to notify the operation and maintenance personnel, and normally stores the detection results in the audit table; and a data security monitoring submodule, which collects network traffic through a mirror port, parses it through Suricata IDS and stores it in Elasticsearch, and stores the detection results in the data storage module. The business filing and supervision module includes: a filing information entry submodule, which automatically loads the corresponding template according to the business type for the business party to fill in online and electronically signs the filing documents; a filing information review submodule, which sets up a three-level review unit to review the filing information; and a filing information tracking submodule, which connects with the business system, automatically collects business operation data on a regular basis, evaluates the consistency between the actual business execution and the filing content, generates a deviation analysis report, and updates the filing information status. The operator supervision module includes: an identity authentication submodule, an operation behavior recording submodule, and an access control submodule; The system and procedure supervision module includes: a system library management submodule, which stores and manages various versions of data supervision systems and procedure documents; a compliance check submodule, which checks whether business operation processes comply with regulations. If non-compliant, it automatically generates a list of compliance defects and a non-compliance report, and links them to the specific responsible department; and a training management submodule, which provides online training courses and assessment functions for operators. The assessment results are linked to the access control module, and those who fail the assessment are automatically restricted from high-risk operation permissions. The data performance monitoring module includes: a data value assessment submodule, which quantitatively assesses the potential economic, strategic, and social value of data resources; a data usage efficiency analysis submodule, which analyzes data usage efficiency through indicators such as access frequency and usage duration of statistical data; and a decision support submodule, which integrates knowledge graph technology to link data assessment results with industry best practice cases and generate decision recommendation documents based on the data value assessment and usage efficiency analysis results, providing decision recommendations for the optimal allocation and utilization of data resources.

2. The big data resource supervision system as described in claim 1, characterized in that, The data collected by the monitoring submodule is compressed using the gRPC protocol and transmitted to the Kafka message queue of the central server's data storage module. The Flink real-time computing platform then consumes the queue data, performs format conversion, and stores it in the ClickHouse database. If the Agent probe program fails to connect to the central server N times consecutively, it automatically switches to offline storage mode, temporarily storing the data in a local SQLite database. The data will be uploaded in batches after the network is restored.

3. The big data resource supervision system as described in claim 1, characterized in that, The LSTM model of the analysis submodule obtains infrastructure operation data from the data storage module and is automatically trained once a day. During the training process of the LSTM model, an Early Stopping strategy is adopted. When the error on the validation set does not decrease for 5 consecutive times, the training is stopped and the optimal model is saved. The prediction model is then updated based on the latest 7 days of data.

4. The big data resource monitoring system as described in claim 3, characterized in that, The analysis submodule includes a device health report generation unit, which automatically generates a device health report if the running device malfunctions.

5. A big data resource monitoring system as described in claim 1, characterized in that, The data quality inspection submodule adopts the DQM framework and uses metadata lineage analysis technology to track the entire lifecycle of data from generation to use. For structured data, it sets rules such as field non-empty checks and value range verification, while for unstructured data, it uses similarity comparison algorithms to detect duplicate content.

6. The big data resource supervision system as described in claim 1, characterized in that, The data security monitoring submodule also includes an anomaly detection unit. When the anomaly detection unit detects frequent access to sensitive data, it immediately triggers the circuit breaker mechanism to block the communication link. When the anomaly detection unit detects unauthorized access attempts, it immediately blocks access and logs the events, while displaying the relevant security events on the visualization interface.

7. A big data resource monitoring system as described in claim 1, characterized in that, The operation behavior recording submodule is built on Apache Kafka to create a sand queue, which captures and records the operation logs of operators on data resources in real time, and realizes centralized storage, retrieval and visualization analysis of logs through ELK technology.

8. A big data resource monitoring system as described in claim 1, characterized in that, The permission management submodule, based on the RBAC model, dynamically allocates or adjusts the scope of data access and operation permissions in real time according to the operator's position, responsibilities, and risk value. If an operator attempts to perform an operation beyond their permissions, the system automatically rejects it and records the log. Alternatively, if the system detects that an operator is accessing sensitive data abnormally frequently, it automatically downgrades their permission level.

9. A big data resource monitoring system as described in claim 1, characterized in that, The visualization interface module includes an interactive unit, through which supervisors can issue operation commands and query data for each module; and uses WebSocket technology to refresh data in real time. When new warning information is generated, the interface automatically pops up a notification box and plays a prompt sound.

Images