Resource pool-oriented information probe acquisition system

Abstract

The invention relates to the technical field of resource pool operation and maintenance management, and particularly discloses a resource pool-oriented information probe acquisition system, which comprises a core function module and a support guarantee module, the support guarantee module is used for formulating an acquisition strategy and standardizing a data format and a core process; and the core function module is used for carrying out data acquisition and data collection on each node in the resource pool according to an acquisition strategy. According to the method, comprehensive and efficient acquisition of related information of various nodes such as physical machines, virtual machines, container instances and edge devices in a resource pool is realized, a transmission-storage-access-operation full-link system is constructed, format consistency, integrity and accuracy of acquired data are ensured, and the data acquisition efficiency is improved. The security isolation and access controllability of different users, tenants and business data are ensured, and the core requirements of fine management, compliance auditing and risk prevention and control of the resource pool are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource pool operation and maintenance management, and more particularly to a resource pool-oriented information probe collection system. BACKGROUND

[0002] With the development of cloud computing, big data and distributed technology, IT systems have gradually evolved from traditional monolithic architecture to intensive architecture with resource pool as the core. Through the unified integration and scheduling of hardware resources (servers, storage devices, network devices), system resources (operating systems, middleware, database systems) and application resources (business applications, tool applications), resource pool realizes dynamic allocation and efficient utilization of resources, and has been widely applied to core business scenarios in government, finance, enterprise and other fields.

[0003] In the process of resource pool operation and maintenance and security protection, information collection is a core foundation link. Only by accurately and comprehensively collecting the attribute information, running state information and correlation information of each level of resource in the resource pool can data support be provided for resource scheduling optimization, fault positioning and security risk identification. At present, there are many monitoring and information collection schemes for resource pool in the industry, but they still have significant defects in architecture design, function coverage and data processing capacity, which cannot meet the needs of fine management of resource pool. The specific problems are as follows: 1) The existing client monitoring scheme focuses on malicious behavior prevention and lacks application white list monitoring capability: The existing client-based monitoring scheme has the core goal of malicious behavior blocking in its architecture design, and the function modules mainly focus on virus detection, online state monitoring, etc. For example, through the real-time virus scanning module, the files and processes in the client are detected for malicious code, through the network connection monitoring module, the network access state and data transmission direction of the client are tracked, and through the process abnormal behavior analysis module, unauthorized process file tampering, registry modification and other operations are identified.

[0004] However, such a scheme has limitations in architecture design, and does not build a white list management system for legal applications in the resource pool. On the one hand, there is no application white list database module, and a standardized legal application list cannot be established based on the unique identifier of the application, which makes it impossible to distinguish the essential difference between unauthorized applications and legal applications. On the other hand, there is no white list application verification module to trace the identity and verify the status of the application running in the client, which leads to the risk that legal applications in the resource pool are replaced by malicious applications and still run normally, and unauthorized applications pretend to be legal applications to access the resource pool.

[0005] 2) The existing monitoring scheme lacks data governance capability, and the collected data is fragmented and non-standardized: In the architecture design of the existing resource pool monitoring scheme, the acquisition module and the storage module are in a linear structure of direct docking, and no independent data governance layer is set, resulting in that the existing scheme can only realize the basic functions of raw data acquisition and storage, and cannot process the unified representation of the collected information. The fine management of the resource pool needs to rely on the standardized association of multi-dimensional information such as application name, version number, country, application category, use location, use unit, user, and use module. The existing scheme has defects such as lack of data cleaning function, insufficient data standardization capability, and weak data association capability.

[0006] 3) The existing monitoring scheme is designed in layers and independently, and cannot realize all-around monitoring of the hardware, system, and application layers. The monitoring modules of the hardware layer, system layer, and application layer belong to different management systems, and the data storage is isolated from each other, which cannot realize end-to-end monitoring and fault tracing of the hardware, system, and application layers, resulting in monitoring blind spots and data islands in the operation and maintenance management of the resource pool, and it is difficult to meet the needs of efficient, safe, and fine operation of the resource pool.

[0007] Therefore, how to integrate application white list monitoring, data governance, and cross-level all-around acquisition function to improve the overall monitoring of the resource pool, data effectiveness, and operation and maintenance efficiency is a problem that needs to be solved by those skilled in the art. SUMMARY

[0008] In view of the above problems, the present application provides an information probe acquisition system for a resource pool to overcome the above problems or at least partially solve the above problems.

[0009] In order to achieve the above purpose, the present application adopts the following technical scheme: An information probe acquisition system for a resource pool, comprising: a core function module and a support and guarantee module; The support and guarantee module is used to formulate an acquisition strategy and standardize data format and core processes; The core function module is used to collect data from each node in the resource pool, unify the format, and aggregate data according to the acquisition strategy.

[0010] Further, the core function module includes a local data acquisition function module, a management agent module, a management service module, and an acquisition and aggregation module; the support and guarantee module includes an acquisition strategy module, a data format management module, and an acquisition interface management module; The acquisition strategy module converts business requirements into machine executable acquisition strategies; The management service module is deployed in the management area of the resource pool, receives business requirements defined by administrators, and distributes the acquisition strategies generated by the acquisition strategy module to the management agent module; The management agent module synchronizes the collection strategy to the local data collection function modules deployed on each node of the resource pool after parsing the collection strategy; The local data collection function module executes the collection task according to the collection strategy and uploads the collected raw data to the management agent module; The management agent module transmits the raw data to the collection and summary module through an encrypted channel after preprocessing the raw data; The collection and summary module verifies and stores the received data according to the specification defined by the data format management module, and uniformly processes various information features in the data according to the pre-constructed data governance system; The collection interface management module is an upper-layer application for standardizing the definition of data query interfaces, realizing the authentication, flow limiting, and monitoring of the interfaces.

[0011] Further, the local data collection function module is deployed on all nodes in the resource pool, supports active collection based on protocols, commands, and APIs, automatically adapts the strategy parameters after receiving the collection strategy, and locally caches the collected data; when collection fails, it retries collection, and if the retry fails, it marks the collection exception and reports it to the management agent module.

[0012] Further, at least two management agent modules are deployed in each resource pool partition, acting as a master module and a backup module respectively; the management agent module includes a data transmission layer, a preprocessing layer, and a state control layer; The data transmission layer is used to establish an encrypted channel and support batch forwarding and incremental forwarding of data; The preprocessing layer is used to perform data cleaning, data correlation, and exception marking on local data; The state control layer is used for heartbeat detection and remote control of the local data collection function module, and automatically switches to the backup module when the master module fails.

[0013] Further, the management service module includes five sub-service layers, namely: a policy management sub-service layer, a node management sub-service layer, an authority management sub-service layer, an alarm management sub-service layer, and a configuration management sub-service layer; The policy management sub-service layer is used to provide a visual interface to define the five elements of the collection strategy, and supports targeted distribution, batch distribution, and incremental distribution of the collection strategy; when the collection strategy is distributed to the corresponding node, the collection strategy is set to take effect immediately or at a specified time, and a strategy pre-check report is generated before the collection strategy takes effect; wherein the five elements of the collection strategy include collection objects, collection indicators, collection frequency, filtering rules, and upload period; The node management sub-service layer is used to realize node registration, node authentication, and node monitoring; The permission management sub-service layer is configured to assign permissions to various user roles based on an RBAC model; The alarm management service layer is configured to set alarm rules according to alarm types, impact ranges, and severity levels; The configuration management sub-service layer is configured to configure global parameters and dedicated parameters of each functional module, and to automatically generate backups after each successful parameter configuration.

[0014] Further, the collection and aggregation module adopts a distributed storage and hierarchical indexing architecture, including a data receiving layer, a data verification layer, a data storage layer, an indexing layer, and a query interface layer; The data receiving layer writes the data uploaded by the management agent module into a distributed message queue for buffering, and then starts multiple threads to process the queue data in parallel, wherein each thread corresponds to a resource pool partition; The data verification layer performs integrity verification, consistency verification, and legality verification on the queue data according to the specifications defined by the data format management module, and temporarily stores the data that fails the verification in an exception data pool; The data storage layer uses multiple storage media to store the verified data in layers according to the access frequency and life cycle of the data, uses Redis cluster storage for real-time data, uses MySQL or PostgreSQL cluster storage for short-term data, and uses HDFS distributed file system for long-term data, and all data are backed up off-site; The indexing layer filters and queries data according to basic dimensions, business dimensions, and technical dimensions; The query interface layer provides two types of standardized data query interfaces and interfaces with the upper layer applications.

[0015] Further, the collection strategy module includes a strategy modeling layer, a strategy optimization layer, and a strategy version management layer; The strategy modeling layer is configured to provide a preset strategy template, modify parameters based on the preset strategy template through a visual drag-and-drop method, or automatically configure strategy parameters according to the application conditions of the collection strategy; The strategy optimization layer is configured to periodically analyze the execution effect of the collection strategy from the perspectives of collection efficiency and resource occupancy, and generate optimization suggestions; The strategy version management layer is configured to record the version number, creator, creation time, and change content of each collection strategy, and supports version backtracking, version comparison, and version archiving.

[0016] Further, the collection interface management module includes an interface definition layer, an interface authentication layer, an interface flow control layer, an interface monitoring layer, and an interface document management layer; The interface definition layer is used to uniformly define the basic information, request specification and response specification of each data query interface; The interface authentication layer is used to verify the validity of the unique certificate held by the current module when calling the internal module interface, and to obtain a temporary Token through a username or password when calling the external data query interface. The API key and Token fields are judged in the interface call. If they do not exist, it is considered as authentication failure; The interface flow limiting layer is used to set a flow limiting threshold for each interface. When the data call volume exceeds the flow limiting threshold, retry is prompted, and the retry time is given in the response header. When the interface call failure rate is greater than or equal to 50%, the fuse mechanism is automatically triggered, and new requests are rejected during the fuse period; The interface monitoring layer is used to monitor the running state of the interface in real time, including the call volume, success rate, response time and error code distribution; The interface document management layer is used to generate interface documents based on preset specifications. The document content includes interface details, debugging tools and version history.

[0017] Further, the data governance system constructs a full-process governance mechanism for recognition, cleaning, standardization, enrichment and quality evaluation according to eight types of information features, including application name, version number, country, application category, use place, use unit, use personnel and use module; In the data collection stage, the eight types of information features are preliminarily automatically recognized. For information features that cannot be automatically recognized, they are marked as to be supplemented, and reported to the management service module through the management agent module for display and supplement; In the data cleaning stage, the recognized information features are cleaned; In the data standardization stage, the cleaned information features are standardized by performing field format unification, classification system unification and coding rule unification according to the specifications defined by the data format management module; In the data enrichment stage, multiple external data sources are connected, and additional information is automatically supplemented to the standardized information features. For additional information that cannot be automatically supplemented, a manual enrichment input interface is provided through the management service module; In the data quality evaluation stage, the data quality evaluation model built in the collection and summary module is used to evaluate the quality of the enriched information features from four dimensions of integrity, accuracy, consistency and timeliness, and divide the quality level, and give the corresponding improvement direction.

[0018] Further, it also includes a security policy system; The security policy system covers the whole link of data transmission, data storage, data access and data operation; In the data transmission stage, the data is fully link encrypted, and for sensitive data, a dynamic encryption key is additionally used for secondary encryption. In the data storage stage, different encryption methods are adopted for structured data, unstructured data and sensitive fields respectively, and backup data is stored in a different place for encryption, when the data reaches the storage life cycle, the data is destroyed by using logical covering or physical destruction, and a data destruction operation record is generated; In the data access stage, data access permissions are set based on user roles and user attributes, and access behavior is controlled; In the data operation stage, audit logs are generated for all key operations, and the audit logs are stored in the blockchain; In the full link stage, all modules are regularly scanned for vulnerabilities and protected, and real-time emergency response strategies are provided.

[0019] According to the technical solution described above, compared with the prior art, the present application has the following beneficial effects: The present application realizes comprehensive and efficient collection of information related to various nodes such as physical machines, virtual machines, container instances and edge devices in the resource pool, constructs a full-link system of transmission-storage-access-operation, ensures the format consistency, integrity and accuracy of collected data, guarantees the safe isolation and controllable access of different users, tenants and business data, meets the core needs of resource pool fine management, compliance audit and risk prevention and control, provides standardized and highly reliable data source support for upper monitoring platform, operation system and the like, and further improves the overall operation efficiency and security protection capability of the resource pool. In addition, the present application provides high-value data support for resource pool optimization configuration and compliance audit through data enrichment (connecting external authoritative data sources) and multi-dimensional index construction. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0021] Fig. 1 The data interaction flowchart of the core function module and support guarantee module provided in the embodiment of the present application; Fig. 2 The structural schematic diagram of the information probe collection system for the resource pool provided in the embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figs. 1-2 As shown, this embodiment of the invention discloses an information probe acquisition system for resource pools, including: a core functional module and a support and guarantee module; The support and assurance module is used to formulate data collection strategies and standardize data formats and core processes, and is responsible for ensuring the standardization, security and reliability of core processes; The core functional modules are used to collect data from various nodes in the resource pool according to the collection strategy, standardize the format, and summarize the data.

[0024] This invention constructs a full-link technology system encompassing acquisition, transmission, management, aggregation, governance, and isolation to achieve efficient, secure, and standardized acquisition and management of application and device information within a resource pool. The core functional modules include a local data acquisition module, a management agent module, a management service module, and an acquisition aggregation module; the supporting and safeguarding modules include an acquisition strategy module, a data format management module, and an acquisition interface management module. The data collection strategy module translates business requirements into machine-executable data collection strategies, which include data collection targets, frequency, and metrics. The management service module is deployed in the management area of ​​the resource pool, receives business requirements defined by the administrator, and distributes the collection policies generated by the collection policy module to the management agent module; After parsing the collection strategy, the management agent module synchronizes it to the local data collection function module deployed on each node of the resource pool; The local data acquisition module executes acquisition tasks according to the acquisition strategy, obtains target information such as application name, version number, and country, and uploads the acquired raw data to the management agent module. After preprocessing the raw data, the management agent module transmits it to the data collection and aggregation module through an encrypted channel; The data collection and aggregation module verifies and stores the received data in accordance with the specifications defined by the data format management module, and performs unified processing on various information features in the data according to the pre-built data governance system. The data acquisition interface management module is for upper-layer applications (such as resource pool monitoring platforms, reporting systems, etc.) and is used to standardize the definition of data query interfaces, and to implement interface authentication, rate limiting and monitoring.

[0025] Next, we will further explain the architecture design and functional implementation of each of the above modules.

[0026] (1) Local data acquisition module: 1) Model Architecture: The local data acquisition module adopts a "lightweight deployment + scalable acquisition engine" architecture, deployed on all nodes within the resource pool, including physical machines, virtual machines, container instances, and edge devices. The module size is ≤50MB, CPU utilization is ≤5%, and memory usage is ≤100MB, avoiding the occupation of core node resources. The module is internally divided into an acquisition engine layer, a data cache layer, and a status monitoring layer. Each layer is decoupled through internal interfaces, supporting independent upgrades.

[0027] 2) Function implementation: After the local data acquisition module receives the acquisition strategy, it automatically adapts the strategy parameters and caches the acquired data locally. When acquisition fails, it retryes the acquisition. If the retry fails, it marks the acquisition as abnormal and reports it to the management agent module.

[0028] ① The acquisition engine layer has multi-mode acquisition capabilities, specifically including: Active Probe Capabilities: Supports active data collection based on protocols, commands, and APIs, including but not limited to: collecting configuration information of network devices (switches, routers) based on the SNMPv3 protocol; collecting application process information at the operating system level based on WMI (Windows) / D-Bus (Linux); executing command-line scripts based on SSH (key authentication) to extract information such as application installation directory and version number; and calling built-in interfaces of applications based on RESTful APIs to obtain structured data such as application category and modules used.

[0029] Passive receiving capability: Capture event information such as application startup / shutdown, version update, and user login in real time through log listening and event subscription.

[0030] ② Data caching layer, which can achieve: Data caching uses a local SQLite database to cache collected data. The cache size is configurable (default 1GB). When the network is interrupted, the cached data can be retained for 72 hours and will be automatically resumed after the network is restored. In case of abnormal retry, such as API call timeout or protocol connection failure, the exponential backoff algorithm is used to retry. The retry interval is 1s→2s→4s→8s, with a maximum of 5 retries. If the retry fails, it is marked as "collection abnormal" and reported to the management agent module.

[0031] (2) Management Agent Module: 1) Model Architecture: The management agent module serves as the intermediate hub between the local data acquisition function module and the management service module. It adopts a primary and backup dual-machine + distributed deployment architecture. At least two management agent modules are deployed in each resource pool partition, serving as the primary module and the backup module respectively. The management agent module includes a data transmission layer, a preprocessing layer and a status control layer, supporting the processing of 1000+ data acquisitions per second with a transmission latency of ≤100ms.

[0032] 2) Functionality Implementation: ① The data transmission layer is used to establish encrypted channels and supports batch and incremental data forwarding; specifically, it is reflected in: Encryption of transmission: The data transmission layer establishes an encrypted channel with the local data acquisition module and the management service module using the TLS 1.3 protocol, and the certificate is signed with the SM2 national cryptographic algorithm to prevent data from being eavesdropped or tampered with; Data forwarding: Supports a combination of batch forwarding and incremental forwarding modes. Regular data is forwarded in batches once every minute, while emergency data, such as data collection anomaly alarm data, is forwarded in real time with incremental forwarding. During the forwarding process, the original data collection timestamp, node ID and other metadata are retained.

[0033] ② The preprocessing layer is used to perform data cleaning, data association, and anomaly marking on local data, specifically in the following ways: Data cleaning: Remove redundant fields, such as duplicate application names and null fields; correct formatting errors, such as unifying "V2.1" to "2.1". Data association: Establish association relationships by combining node ID and application process ID to avoid the same application being counted repeatedly on different nodes; Anomaly marking: For abnormal data that does not conform to the collection strategy, such as incorrect version number format or invalid country code, mark it as "pending review" and attach the reason for the anomaly, such as the version number not conforming to the semantic specification; ③ The status control layer is used for heartbeat detection and remote control of the local data acquisition module. When the main module fails, it automatically switches to the backup module. Specifically, this is reflected in: Health monitoring: Periodically send heartbeat detection packets to the local acquisition module to monitor the module process's survival status and CPU / memory usage. If no heartbeat response is received for 3 consecutive times, the acquisition node is determined to be offline. Remote control: Supports issuing remote commands through the management service module to perform operations such as restarting, upgrading, and updating configurations on the local acquisition module, and operation logs are synchronized to the audit system in real time; Fault self-healing: When the main module fails, the backup module automatically switches over within 10 seconds. During the switching process, data transmission is not interrupted based on session persistence technology, ensuring that the acquisition link is unobstructed.

[0034] (3) Management Service Module: 1) Module Architecture: The management service module serves as the core control hub of the entire system. It adopts a combination of microservice architecture and cluster deployment, and includes five sub-service layers: policy management sub-service layer, node management sub-service layer, permission management sub-service layer, alarm management sub-service layer, and configuration management sub-service layer. It is deployed in the management area of ​​the resource pool, supports horizontal scaling, and a single cluster can support up to 10,000+ collection nodes with service availability ≥99.99%.

[0035] 2) Functionality Implementation: ① The strategy management sub-service layer provides a visual interface for defining the five elements of a data collection strategy. It supports targeted, batch, and incremental distribution of strategies. Targeted distribution applies only to specified nodes, batch distribution applies to groups of nodes, and incremental distribution updates only the changed parts of the strategy. The distribution results are fed back in real time, including messages such as "** nodes have received the strategy, * nodes are offline and awaiting retry." After a data collection strategy is distributed to the corresponding nodes, it is set to take effect immediately or on a scheduled basis. A strategy pre-verification report is generated before the strategy takes effect. Immediate effect means execution immediately after distribution, while scheduled effect means it takes effect at a specific time to avoid impacting peak business periods. The strategy pre-verification report includes information such as the number of nodes covered by the strategy and the compatibility of data collection metrics.

[0036] The five key elements of the data collection strategy include the collection target, collection metrics, collection frequency, filtering rules, and upload cycle. The collection target refers to the specified range of resource pool nodes, such as all physical machines in the Beijing partition or the container cluster in the Shanghai partition. Collection metrics refer to selecting target information characteristics, such as application name, version number, country of origin, and user organization. Collection frequency refers to setting a fixed or dynamic frequency, such as every 5 minutes when resource load is <50%, and every 20 minutes when load is ≥50%. Filtering rules refer to configuring a whitelist or blacklist; a whitelist can be used to collect only Office and MySQL applications; a blacklist can be used to exclude temporary test applications. The upload cycle refers to setting the period at which data is uploaded to the data collection and aggregation module.

[0037] ② The node management sub-service layer is used to implement node registration, node authentication, and node monitoring; The node registration process is as follows: When a data collection node is deployed for the first time, it automatically sends a registration request to the management service module, carrying information such as node ID, IP address, hardware configuration (number of CPU cores, memory capacity), partition to which it belongs, and operating system type. Registration is completed after the administrator approves the request.

[0038] The node authentication process is as follows: a two-factor authentication method using certificate authentication and dynamic token is adopted. Each time a node connects, it must submit a dynamic token with a validity period of 1 hour. This token is issued periodically by the management service module to prevent unauthorized nodes from accessing the network. The node monitoring process involves: real-time display of the node's online status, data collection task execution progress, and resource utilization; support for filtering by partition, node type, data collection status, etc.; and automatic marking and triggering of alarms for offline nodes.

[0039] ③ The permission management sub-service layer is used to assign permissions to various user roles based on the RBAC model; specifically, it includes: role definition, permission mapping, and data permissions; role definition refers to: four preset roles, namely system administrator, policy administrator, auditor, and operator, and supports custom roles; permission mapping refers to: each role is bound to specific permissions, for example, the policy administrator has policy creation / modification / deletion permissions, and the auditor only has log query / report export permissions; data permissions refer to: restricting users to access only the data of their respective partition / business, for example, the operator of the Beijing partition cannot view the collected data of the Shanghai partition, and permission change operations are recorded in the audit log and cannot be tampered with.

[0040] ④ The alarm management service layer is used to set alarm rules according to alarm type, scope of impact, and severity. Alarm types include node offline, data collection failure, and data anomaly; scope of impact includes single node, multiple nodes, or the entire partition; severity includes urgent, important, general, and alert. Alarm rules include three scenarios: alarm triggering, alarm notification, and alarm closure. The alarm triggering rule is: when the rule conditions are met, such as when ≥5 nodes in a single partition are simultaneously offline, an alarm is automatically triggered. The alarm notification rule supports email, SMS, and API integration with third-party monitoring platforms. The notification content includes the alarm type, trigger time, affected nodes, and handling suggestions. The alarm closure rule supports a closed-loop process of alarm reception → handling assignment → handling feedback → alarm clearing. Alarms not handled within a preset time period are automatically escalated to notify the superior administrator.

[0041] ⑤ The configuration management sub-service layer is used to configure global parameters and dedicated parameters for each functional module. It automatically generates backups after each successful parameter configuration and supports rolling back to historical configurations to prevent system failures caused by configuration errors. Global parameters include data transmission timeout, cache capacity limit, and certificate validity period, while dedicated parameters include the CPU usage limit for the local data acquisition module and the retry count for the management agent module. (4) Data Collection and Summary Module: 1) Module Architecture: The data collection and aggregation module serves as the data storage and query center, adopting a distributed storage and hierarchical index architecture, including a data receiving layer, a data verification layer, a data storage layer, an index layer, and a query interface layer; it supports receiving 5000+ data entries per second, and the storage capacity of a single cluster can reach the PB level. For single-condition queries, the query response time is ≤1 second, and for multi-condition combined queries, the query response time is ≤3 seconds.

[0042] 2) Functionality Implementation: ① The data receiving layer first writes the data uploaded by the management agent module into the distributed message queue Kafka for buffering, and then starts multiple threads to process the queue data in parallel. Each thread corresponds to a resource pool partition, and there are up to 10 threads to achieve partition isolation and parallel processing, and monitor the consumption progress in real time. ② The data validation layer performs integrity, consistency, and validity checks on queue data according to the specifications defined by the data format management module. Data that fails the checks is temporarily stored in the abnormal data pool, allowing administrators to view details and trigger re-collection. Integrity checks include verifying the presence of all required fields, such as application name, version number, and collection timestamp; missing fields are marked as incomplete. Consistency checks verify that the data format conforms to specifications, such as whether the version number is in "XYZ" format and whether the country code is ISO 3166-1 alpha-2; incorrect formats are marked as inconsistent. Validity checks verify that data values ​​are within a reasonable range, such as whether the user account conforms to the department abbreviation + employee number rule and whether the application category is within the preset enumeration value; invalid values ​​are marked as invalid.

[0043] ③ The data storage layer uses multiple storage media to stratify and store verified data according to access frequency and lifecycle. Specifically, a Redis cluster is used to store real-time data within 24 hours, supporting high-frequency queries and automatically deleting expired data; a MySQL or PostgreSQL cluster is used to store short-term structured data within the last 3 months, supporting transaction consistency and SQL queries; and an HDFS distributed file system is used to store long-term data (over 1 year), including raw data and governance logs, partitioned by "year and month," supporting low-cost archiving. All data is backed up using an off-site backup strategy, and backup data is encrypted using the AES-256 algorithm, with daily incremental backups and weekly full backups.

[0044] ④ The index layer is built on Elasticsearch to create a full-text search index, enabling multi-dimensional data indexing. Specifically, data is filtered and queried according to basic dimensions, business dimensions, and technical dimensions. Basic dimensions include: application name, version number, country, and application category; business dimensions include usage location, user organization, user, and module used; technical dimensions include collection node ID, collection timestamp, and data quality level. The index supports fuzzy queries and range queries.

[0045] ⑤ The query interface layer provides two types of standardized data query interfaces and connects to upper-layer applications. These are a RESTful API and an SQL query interface. The RESTful API returns data in JSON format, supports single and batch queries, and uses the unified prefix ` / api / v1 / collect / data` in its address. Request parameters include query dimensions, filter conditions, and pagination information. The returned results include a list of data, total number of records, and pagination identifier. The SQL query interface is compatible with standard SQL syntax, supports JDBC / ODBC connections, and only allows SELECT statements, prohibiting write operations such as UPDATE / DELETE / ALTER to prevent data tampering. Both types of interfaces use API Key+Token dual authentication and support rate limiting based on a token bucket algorithm, such as a maximum of 100 calls per second for a single interface. Interface call logs include the caller, call time, request parameters, and response results, and are written to the auditing system in real time.

[0046] (5) The data collection strategy module is the strategy definition and management engine of the management service module. It is responsible for transforming the administrator's business requirements into machine-executable data collection rules. It includes a strategy modeling layer, a strategy optimization layer, and a strategy version management layer. The specific implementation details are as follows: ① The strategy modeling layer is used to provide preset strategy templates, including core application acquisition templates, general application acquisition templates, and test application acquisition templates. Parameters can be modified based on the preset strategy templates through a visual drag-and-drop method, without having to build from scratch; or the strategy parameters can be automatically configured according to the application conditions of the acquisition strategy. For example, when the application category is industrial control software, the acquisition frequency is set to every 5 minutes, otherwise it is set to every 30 minutes.

[0047] ② The strategy optimization layer has a built-in strategy optimization engine, which periodically analyzes the execution effect of collection strategies from the perspectives of collection efficiency and resource utilization, and generates optimization suggestions. The optimization suggestions can be output in the form of a strategy optimization report, which includes a list of strategies to be optimized, optimization schemes, expected effects, etc. After the administrator confirms, the optimization can be automatically applied. Among them, the strategy execution effect includes collection efficiency analysis and resource utilization analysis. The collection efficiency analysis method is to count the collection success rate and data integrity. If the collection success rate of a certain strategy is <90%, it will automatically suggest adjusting the collection protocol. The resource utilization analysis method is to count the CPU / memory utilization of the collection task on the node. If the utilization rate is ≥10%, it will automatically suggest reducing the collection frequency.

[0048] ③ The strategy version management layer records the version number, creator, creation time, and changes for each data collection strategy, and supports version rollback, version comparison, and version archiving. Version rollback refers to reverting to any historical version to avoid data collection anomalies caused by the current strategy; version comparison allows for visual comparison of differences between different versions; and version archiving involves archiving unused strategy versions, which cannot be modified after archiving and can only be viewed.

[0049] (6) Data format management module: responsible for defining the unified format specifications of collected data, ensuring that the data structure of collected data from different nodes and applications is consistent, and providing a foundation for subsequent data governance and query. Core functions include format definition, format verification, and format compatibility.

[0050] ① Unified format definition: For the eight types of information features that need to be collected in this invention, strict format specifications are defined, as shown in Table 1.

[0051] Table 1 Format Definition

[0052] ② Format validation and conversion: This function is implemented by providing a format validation interface. After receiving the data, the data collection and aggregation module calls this interface to perform the following validations: Automatic validation: Automatically checks whether the data conforms to the format specifications through regular expressions, enumeration matching, etc. Automatic conversion: Automatically converts data with minor formatting errors to a standard format; Manual intervention: For data that cannot be automatically converted, such as "Version number 1.2.3.4" which does not conform to semantic markup, mark it for manual correction and notify the administrator to enter the correct format through the management service module.

[0053] ③ Format version compatibility: When the format specification needs to be changed, such as adding a "Application Deployment Environment" field, the following backward compatibility is supported: Version identifier: A version number is assigned to each format specification, and a new "Format Version" field is added to the data; Format conversion: After uploading old version data, the format conversion tool will be automatically called to convert it to the current version. Missing new fields are marked as to be added later. Compatibility period: The old version format will be retained for 180 days. During this period, both the old and new versions of data can coexist. After the expiration date, the old version data will be automatically stopped being received. You will need to convert to the new version first.

[0054] (7) The data acquisition interface management module is responsible for the standardization, security, and lifecycle management of all system interfaces, ensuring the stability and maintainability of the interfaces. Its core functions include the interface definition layer, interface authentication layer, interface rate limiting layer, interface monitoring layer, and interface documentation management layer. The specific implementation details are as follows: ① The interface definition layer is used to uniformly define the metadata of each data query interface, including basic information, request specifications, and response specifications.

[0055] Basic information includes: interface name, unique identifier API ID, module to which it belongs, and interface type (internal interface / external interface).

[0056] The request specifications include: request method (GET / POST / PUT / DELETE), request URL, request parameters (name, type, required, default value), and request headers.

[0057] The response specifications include: response status codes, such as 200 = success, 400 = parameter error, 401 = unauthenticated, 500 = server error, etc.; response body format; and response fields (name, type, description).

[0058] The following example illustrates this: The request parameters for the "Query Application Information Interface" include "Application Name (required), Country (optional), Page Number (default 1), Page Size (default 20)", and the response body includes "code (status code), msg (prompt message), data (data list), total (total number of entries)".

[0059] ② The interface authentication layer is used to authenticate all interfaces (including interfaces between internal modules and external query interfaces). When calling interfaces between internal modules, it verifies the validity of the unique certificate held by the current module to prevent unauthorized modules from accessing the interface. When calling external data query interfaces, it first obtains a temporary token using a username or password. When calling the interface, it checks whether the request header carries the API key + Token field. If not, authentication is considered to have failed, and a 401 error message is returned.

[0060] ③ The interface rate limiting layer uses a token bucket algorithm and a circuit breaker mechanism to prevent excessive calls to the interface from causing system overload. Specifically: Rate limiting configuration: Set rate limiting thresholds for each interface, such as a maximum of 100 calls per second for the query interface and a maximum of 10 calls per second for the batch import interface. The thresholds can be adjusted at the user / tenant level, and the thresholds for administrator accounts are higher than those for ordinary users.

[0061] Rate limiting response: When the data call volume exceeds the rate limiting threshold, a 429 error message is returned, prompting for a retry, and the retry time is given in the response header.

[0062] Circuit breaker mechanism: When the failure rate of interface calls is ≥50%, such as when a database failure causes a query failure, the circuit breaker mechanism is automatically triggered, and new requests are rejected during the circuit breaker period, and a 503 error is returned directly to prevent the failure from spreading.

[0063] ④ The interface monitoring layer is used to monitor the running status of interfaces in real time, including call volume, success rate, response time, and error code distribution. The monitoring data is displayed in real time on the interface monitoring dashboard in the management service module, and supports the generation of daily / weekly reports, including "interface running trends, anomaly analysis, and optimization suggestions". Call volume is defined as the number of calls per second / minute / hour, and statistics can be performed by interface, user, and IP. Success rate is defined as the number of successful calls / total number of calls. An alarm is triggered if the success rate is less than 95%. Response time is defined as the average response time, maximum response time, and P95 response time (95% of request response times are ≤ this value). An alarm is triggered if the response time is greater than 3 seconds. Error code distribution is defined as the number of occurrences of each error code, with high-frequency error codes triggering alarms.

[0064] ⑤ The API documentation management layer is used to generate API documentation based on preset specifications. The documentation supports export in HTML, PDF, and Markdown formats for use by developers and stakeholders. Updates to the documentation automatically notify relevant personnel. The documentation content includes API details, debugging tools, and version history. API details include: request / response specifications, parameter descriptions, and example requests / responses. The debugging tools support online debugging; users can input parameters, send requests directly, and view the response results. Version history records the document's update time, updated content, and the person who updated it.

[0065] This invention relates to three systems: a security strategy system, a data governance system, and a data isolation system. The three systems are further explained below: (8) The security strategy system covers the entire chain of data transmission, data storage, data access, and data operation; it constructs a multi-layered, defense-in-depth security mechanism, the specific implementation process of which includes: 1) During the data transmission phase, the data is encrypted across the entire link. For sensitive data, a second encryption is performed using a dynamic encryption key.

[0066] ① End-to-end encryption: The local data acquisition module and the management agent module are encrypted using the TLS 1.3 protocol, with the key exchange algorithm being ECDHE, the symmetric encryption algorithm being AES-256-GCM, and the hash algorithm being SHA-384, to prevent data from being eavesdropped on or tampered with. The management agent module, management service module, and data collection and aggregation module use IPsec VPN to establish a dedicated encrypted channel. The tunnel encryption algorithm is AES-256, and the authentication algorithm is HMAC-SHA-256 to ensure secure cross-regional transmission. External query interface: HTTPS protocol is used, and certificates are issued by nationally recognized CA institutions. Self-signed certificates are prohibited.

[0067] ② Additional protection for sensitive data: Sensitive data such as user identity information, application license keys, and database account passwords are encrypted using the SM4 national cryptographic algorithm before transmission. The encryption key is dynamically distributed by the management service module and updated once every 24 hours. Only the receiving module can decrypt it, preventing key leakage from exposing sensitive data.

[0068] 2) During the data storage phase, different encryption methods are used for structured data, unstructured data, and sensitive fields. Backup data is also stored in an encrypted off-site location. When data reaches the end of its storage lifecycle, it is destroyed using either logical overwrite or physical destruction, and a data destruction operation record is generated. Specifically: ①Storage encryption: Structured data (MySQL / PostgreSQL): Transparent Data Encryption (TDE) technology is used to encrypt database files in real time. The key is stored in a separate Key Management Server (KMS). The KMS adopts a dual-machine hot standby architecture to prevent key loss.

[0069] Unstructured data (HDFS / log files): File-level encryption is used. Each file generates a unique encryption key, which is stored in the metadata file after encryption. Decryption requires obtaining the master key from KMS. Sensitive fields: Sensitive fields such as user names and contact information of user organizations are encrypted at the field level (AES-256). When querying, they need to be decrypted before being returned. The encryption / decryption operation is automatically completed by the database stored procedure and does not require application layer processing.

[0070] ② Data backup and destruction: Backup strategy: Daily incremental backup and weekly full backup strategy are adopted. Backup data is stored with encryption (AES-256). Backup media are stored off-site, for example, the main storage is in Beijing and the backup storage is in Shanghai. The backup retention period is 1 year. Destruction Strategy: When data reaches the end of its storage lifecycle or is no longer needed, a combination of logical overwriting and physical destruction is employed. Logical destruction involves performing three overwrite operations on the data in the storage medium: first writing 0, then 1, and finally a random number. Physical destruction involves physically shredding or degaussing the hard drive or server to ensure the data is unrecoverable. Destruction Audit: All data destruction operations are recorded in the "Data Destruction Log", which includes "data type destroyed, destruction time, destruction method, and executor". The log is retained for 3 years.

[0071] 3) During the data access phase, data access permissions are set based on user roles and user attributes, and access behavior is controlled. Specifically, this is reflected in: ① Fine-grained access control: Integrating RBAC and ABAC (attribute-based access control) models to achieve dual access control based on roles and attributes. RBAC refers to assigning basic permissions based on roles, such as operators being able to query data and administrators being able to modify policies. ABAC refers to defining access rules based on "user attributes (company affiliation, role), data attributes (user organization, country), and environment attributes (access IP, access time)." For example, only operators in the Beijing zone are allowed to access application data in the Beijing zone during working hours (9:00-18:00); only administrators are allowed to access sensitive data from the company's intranet IP (192.168.0.0 / 16).

[0072] ② Access behavior control involves login security control, session security control, and privilege escalation protection control. Login security control supports two-factor login with password and verification code. Passwords must be at least 8 characters long, contain uppercase and lowercase letters, numbers, and special characters, and are valid for 90 days. After 5 consecutive failed login attempts, the account is locked for 30 minutes. Session security control generates a unique session ID after user login. The session is valid for 30 minutes and supports forced session termination. Administrators can remotely terminate abnormal sessions. Privilege escalation protection involves performing permission verification on all access requests. If a user attempts to access data without permission, such as a Shanghai partition operator querying Beijing partition data, the access is immediately denied, and an privilege escalation log is recorded, triggering an alarm.

[0073] 4) During the data operation phase, audit logs are generated for all key operations and stored on the blockchain, specifically: ① Full-link audit logs: Audit logs are generated for all critical operations. The logs include the operator, account, time, type, content, client IP, device information, and result. Operations requiring auditing include: System operations: User login / logout, account creation / modification / deletion, permission changes; Strategy operations: creating / modifying / deleting / issuing data collection strategies; Data operations: data query / export / modification / deletion, and correction of abnormal data; Configuration operations: system parameter modification, module configuration change, interface rate limiting adjustment.

[0074] ② The audit logs are protected and are characterized as unalterable, undeletable, and traceable.

[0075] The immutability is achieved by storing audit logs using blockchain technology, implemented through a consortium blockchain. The nodes include a management service module, a data collection and aggregation module, and third-party audit nodes. A block is generated every 10 minutes, and each block contains the hash value of the previous block, ensuring that the logs cannot be tampered with.

[0076] The basis for making logs non-deletable is that audit logs must be retained for at least 6 months to meet the requirements. After the retention period expires, they will be automatically archived to offline storage and cannot be deleted after archiving.

[0077] The traceability is achieved by supporting the querying of audit logs by operator, operation time, operation type, and other dimensions, and supporting log export (PDF / Excel format) for compliance checks and incident tracing.

[0078] 5) Vulnerability Protection and Emergency Response: Throughout the entire process, all modules are regularly scanned for vulnerabilities and protected, and emergency response strategies are provided in real time.

[0079] ①The specific implementation process of vulnerability protection and safeguards is as follows: Regular scanning: Monthly vulnerability scans of all modules are performed using automated vulnerability scanning tools, and penetration tests are conducted quarterly by a third-party security company; Secure coding: During the development phase, we follow the OWASP Top 10 secure coding standards to prevent common vulnerabilities such as SQL injection, XSS, and CSRF. Static code scanning is performed before code submission. Patch management: Security patches for operating systems, databases, and middleware are updated within 72 hours of release, and compatibility testing is performed before updates to avoid affecting system operation; Boundary protection: Deploy a next-generation firewall (NGFW) and intrusion detection system (IDS) at the front end of the management service module and data collection and aggregation module to block malicious IPs, SQL injection attacks, and DDoS attacks.

[0080] ②The specific implementation process of the emergency response is as follows: Emergency Response Plan: Develop an emergency response plan for information security incidents, clearly defining incident levels, with Level 1 being core data leakage / system paralysis; Level 2 being partial functional failure / data anomaly; and Level 3 being minor alarms / single node offline. Then, develop a response process based on the time level: discovery → reporting → analysis → handling → recovery → summary, and assign responsibilities, and implement emergency team notification, incident handling, isolation, loss mitigation, and recovery measures.

[0081] The emergency response team is generated as follows: an emergency response team is established, including system administrators, security engineers, and development engineers, on standby 24 / 7.

[0082] The incident handling procedure is as follows: When a security incident occurs, the emergency response plan will be activated immediately. Isolation measures include suspending the affected modules, such as closing vulnerable interfaces and isolating infected data collection nodes.

[0083] The measures to mitigate the damage are as follows: If it is a data breach, immediately shut down the source of the breach, such as taking the illegal query interface offline and notifying the relevant users to change their passwords.

[0084] The recovery measures are as follows: restore the damaged system by backing up the data, fix the vulnerabilities, and then put it back online.

[0085] (9) The data governance system constructs a full-process governance mechanism for identification, cleaning, standardization, enrichment, and quality assessment based on eight information characteristics: application name, version number, country of origin, application category, location of use, user unit, user, and module used. This mechanism ensures the integrity, accuracy, consistency, and timeliness of the collected data. The specific implementation method is as follows: 1) During the data collection phase, the eight types of information features are initially automatically identified. For information features that cannot be automatically identified, they are marked as to be supplemented and reported to the management service module through the management agent module for display and supplementation.

[0086] ①Automatic recognition: The application name and version number are obtained by parsing the application configuration file, the installation directory file name, and the results returned by the application's built-in API. For the country of origin, it is obtained by parsing the application license file (containing regional information), the language settings of the installation package, and the regional affiliation of the application vendor; For application categories, the system automatically classifies various common applications by matching the application category knowledge base, such as MySQL → Cloud Computing and Big Data Software - Database Software - Relational Database. For the usage location, it is obtained through the IP address location of the collection node (connected to the IP2Location database), GPS positioning (supported by edge devices), and device deployment location configuration (preset by the administrator as "Beijing Data Center"); For user organizations, this information is obtained through the application's registration information and the organization to which the data collection node belongs; For users, this information is obtained through the application's login logs and user association configurations. For modules used, they can be obtained through the application's process list or the application's built-in module list.

[0087] ② Manual Supplementation: For features that cannot be automatically identified, such as old applications without version numbers or custom applications without category mapping, the local data collection module marks them as needing supplementation and reports them to the management service module through the management agent module. The management service module provides a visual supplementation interface, displaying a list of data to be supplemented, including collection nodes, application names, and missing features; and performs batch supplementation, supporting batch supplementation by importing Excel files; after manual supplementation, the system automatically verifies the format, updates the data after successful verification, and prompts the error reason if verification fails.

[0088] 2) During the data cleaning stage, the identified information features are cleaned by removing redundancy, correcting errors, and merging duplicates.

[0089] When removing redundancy, for field redundancy, duplicate fields are deleted. For example, if the application name of the same application appears twice in the collection results, only one instance is retained. Null value fields are deleted. For example, if the module used is empty and cannot be filled, it is marked and then deleted. For content redundancy, redundant characters in the field content are removed.

[0090] When correcting errors, minor formatting errors are automatically corrected, such as "Version number v2.1" → "2.1", "Country of origin China" → "CN", and "Usage location Beijing" → "China-Beijing-Unknown location type". For logical errors, contradictory data is corrected, such as if the application category is office software but the usage location is a production workshop. This requires administrator confirmation before correction, and invalid data is also corrected.

[0091] When merging duplicate data, duplicate data collected for the same application on different nodes should be merged according to the following rules: Complete duplicates: If all fields of the data are identical, keep one record and delete the rest; Partial duplication: Core fields (application name, version number, user unit) are the same, while other fields (user, module used) are different. These are merged into one data entry, with different fields separated by "|" (e.g., "user: Zhang San|Li Si"). Suspected Duplicate: Applications with similar names, such as "Office 2019" and "Microsoft Office 2019", are marked as suspected duplicates if the similarity is ≥90% using a string similarity algorithm (such as Levenshtein distance). The administrator will then confirm whether to merge them.

[0092] 3) During the data standardization phase, the cleaned information features are standardized in terms of field format, classification system, and coding rules, in accordance with the specifications defined by the data format management module.

[0093] When standardizing the field format, the format specifications in Table 1 were strictly followed, and the format of the eight types of information features was converted. The application name was standardized to the format "vendor-product-series"; the version number was standardized to the semantic versioning format; the country was standardized to ISO 3166-1 alpha-2 code, such as "United States" → "US", "Japan" → "JP"; the application category was standardized to a three-level classification code, such as "MySQL" → "CBS-DBS-RDBS" (Cloud Computing and Big Data Software - Database Software - Relational Database); the usage location was standardized to the format of country-province-city-location type; the user unit was standardized to the format of organization type-name-department; the user was standardized to the format of account-name-department; and the module was standardized to the format of module code-name-function.

[0094] When implementing a unified classification system, three standardized classification systems are established: an application category classification system, an organization type classification system, and a location type classification system. All data strictly adheres to these system classifications. The application category classification system is divided into three levels. The first level includes six categories: Office and Productivity Software (OPS), Industrial Control and Automation Software (ICS), Cloud Computing and Big Data Software (CBS), Network and Security Software (NSS), Entertainment and Multimedia Software (EMS), and Other Software (OTS). The second level includes 32 categories, including OPS (with subcategories such as WPS, ESS, and PPS). The third level includes 128 categories, including WPS (with subcategories such as LWPS and OWPS). The organization type classification system includes five categories: enterprises, government agencies, public institutions, research institutes, and individuals. The location type classification system includes eight categories: office buildings, data centers, production workshops, laboratories, homes, mobile devices, branch offices, and others.

[0095] When implementing unified encoding rules, a unique code is assigned to key information features, and the encoding rules are as follows: For application coding, the format is: manufacturer abbreviation-product abbreviation-version number; For application category coding, the format is: Level 1 category code - Level 2 category code - Level 3 category code; For module coding, the format is application code - module number; For node encoding, the format is partition abbreviation-node type-serial number.

[0096] 4) During the data enrichment stage, multiple external data sources are connected to automatically supplement the standardized information features with additional information. For additional information that cannot be automatically supplemented, a manual enrichment and input interface is provided through the management service module. The collection and aggregation module supplements the standardized data with additional information through external data source connection and manual enrichment, thereby enhancing the data value.

[0097] ① It connects to multiple authoritative external data sources and automatically supplements information. For application vendor databases, it connects to the official databases of vendors such as Microsoft, Alibaba, and Huawei to supplement information such as application release time, supported operating systems, and vendor contact information. For industry classification databases, it connects to the National Bureau of Statistics database to supplement information such as the industry code and enterprise size (large / medium / small) of the user unit. For IP address geolocation databases, it connects to IP address databases to supplement information such as the detailed address (street, house number) and carrier information of the usage location. For semantic version databases, it verifies the compliance of version numbers and supplements version update logs.

[0098] During integration, scheduled synchronization (full data synchronization every morning at midnight) ensures data timeliness, while real-time querying (interface querying when supplementing) ensures data accuracy.

[0099] ② For information that cannot be supplemented through external data sources, such as customized application functions or the internal departmental structure of the user unit, the administrator enters the information through the manual enrichment interface of the management service module and performs enrichment task allocation, enrichment progress tracking, and enrichment result verification operations. Specifically, during enrichment task allocation, the system assigns enrichment tasks to corresponding administrators according to region and business line. Each task includes a list of data to be enriched, fields to be supplemented, and supplementary instructions. During enrichment progress tracking, administrators can view task progress, and the system sends reminders for tasks that have not been completed within the time limit. During enrichment result verification, the system automatically verifies the format of the supplemented information; if verification passes, the data is updated; otherwise, it returns for modification.

[0100] 5) During the data quality assessment phase, the data quality assessment model is built into the data collection and aggregation module to evaluate the quality of the enriched information features from four dimensions: completeness, accuracy, consistency, and timeliness. The quality levels are then classified, and corresponding improvement directions are given.

[0101] The completeness assessment accounts for 30 points, mainly assessing whether the 8 types of information features are complete. It can be judged based on the absence of "to be supplemented" markers and whether the enriched information is complete. 3.75 points are deducted for each missing feature and 1 point is deducted for each missing enriched information.

[0102] Accuracy assessment accounts for 30 points, mainly evaluating whether the format conforms to the specifications, whether the data values ​​are legal, and whether the enriched information is accurate; 2 points are deducted for each format error, 3 points are deducted for each illegal value, and 2 points are deducted for each enriched information error.

[0103] Consistency assessment accounts for 20 points, mainly assessing whether the data of the same application is consistent across different nodes and whether the classification codes match the system; 2 points are deducted for each data conflict between nodes and 2 points are deducted for each mismatch in classification codes.

[0104] Timeliness assessment accounts for 20 points, mainly evaluating whether the data collection time is within the period specified by the strategy and whether the data is up-to-date; 1 point is deducted for every 10 minutes of collection delay, and 2 points are deducted for each piece of expired data.

[0105] The data is divided into four quality levels based on the total score. Data with a score of ≥90 is considered high-quality and can be directly used for upper-level applications such as monitoring and reporting. Data with a score of 80 ≤ <90 is considered good and requires minor correction before use. Data with a score of 60 ≤ <80 is considered acceptable and requires significant correction before use. Data with a score <60 is considered unacceptable and requires re-collection or manual repair.

[0106] Afterwards, quality improvements are implemented. For minor issues with good data, the system automatically attempts to supplement through external data sources to achieve automatic repair. For format errors in qualified data, the system automatically calls a format conversion tool to correct them. For unqualified data, the system automatically notifies the management agent module to re-collect the corresponding data. If the data is still unqualified after re-collection, it is marked for manual repair. Administrators can view the details of the unqualified data through the data repair interface, manually correct it, and then re-evaluate it.

[0107] After the data quality assessment and correction are completed, a data quality report is generated weekly. The report includes: quality level distribution and problem analysis. The quality level distribution shows the percentage of data at different levels, and the problem analysis shows the main reasons for the non-compliant data.

[0108] This invention's entire system is based on a designed adaptation and optimization knowledge base, which includes a performance parameter library, a performance indicator library, a performance tuning target library, a hotspot / bottleneck judgment rule library, a reference optimization scheme library, and a performance optimization rule library. Then, it generates recommended optimization schemes using optimization rule sets that match performance bottleneck types and adaptation targets. The system's internal databases include a software management library, a license management library, a software tracking library, an application management library, and a user database. Specifically, the software management library manages software file data, operational data, approval process data, and linear management data; the license management library includes application information data, permission management data, license allocation data, license statistics data, license status data, and license file data; the software tracking library includes user information records, download statistics data, usage record data, system access data, software status data, and online user information; the application management library includes release data, application data, trial data, and recommendation data; and the user database includes user information, permission management data, log management data, and department management data.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0110] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An information probe acquisition system for resource pools, characterized in that, include: Core functional modules and supporting modules; The support and assurance module is used to formulate data collection strategies and standardize data formats and core processes; The core functional module is used to collect data from each node in the resource pool, standardize the format, and summarize the data according to the collection strategy.

2. The information probe acquisition system for resource pools as described in claim 1, characterized in that, The core functional modules include a local data acquisition module, a management agent module, a management service module, and an acquisition and aggregation module; the support and guarantee modules include an acquisition strategy module, a data format management module, and an acquisition interface management module. The data acquisition strategy module translates business requirements into machine-executable data acquisition strategies. The management service module is deployed in the management area of ​​the resource pool, receives business requirements defined by the administrator, and distributes the collection policies generated by the collection policy module to the management agent module. After parsing the collection strategy, the management agent module synchronizes it to the local data collection function module deployed on each node of the resource pool. The local data acquisition module executes acquisition tasks according to the acquisition strategy and uploads the acquired raw data to the management agent module; After preprocessing the raw data, the management agent module transmits it to the data collection and aggregation module through an encrypted channel; The data collection and aggregation module verifies and stores the received data in accordance with the specifications defined by the data format management module, and performs unified processing on various information features in the data according to the pre-built data governance system. The data acquisition interface management module is an upper-layer application used to standardize the definition of the data query interface and implement interface authentication, rate limiting, and monitoring.

3. The information probe acquisition system for resource pools as described in claim 2, characterized in that, The local data acquisition module is deployed on all nodes in the resource pool. It supports active acquisition based on protocols, commands, and APIs. When a acquisition strategy is received, it automatically adapts the strategy parameters and caches the acquired data locally. If data collection fails, a retry is performed. If the retry also fails, the data collection is marked as abnormal and reported to the management agent module.

4. The information probe acquisition system for resource pools as described in claim 2, characterized in that, At least two of the aforementioned management agent modules are deployed in each resource pool partition, serving as the primary module and the backup module, respectively; the management agent module includes a data transmission layer, a preprocessing layer, and a state control layer; The data transmission layer is used to establish encrypted channels and supports batch forwarding and incremental forwarding of data; The preprocessing layer is used to perform data cleaning, data association, and anomaly marking on local data; The status control layer is used to perform heartbeat detection and remote control of the local data acquisition function module. When the main module fails, it automatically switches to the backup module.

5. The information probe acquisition system for resource pools as described in claim 2, characterized in that, The management service module comprises five sub-service layers: policy management sub-service layer, node management sub-service layer, permission management sub-service layer, alarm management sub-service layer, and configuration management sub-service layer. The strategy management sub-service layer provides a visual interface for defining the five elements of a collection strategy and supports targeted, batch, and incremental distribution of collection strategies. After a collection strategy is distributed to the corresponding node, it is set to take effect immediately or on a timed basis, and a strategy pre-verification report is generated before the collection strategy takes effect. The five elements of a collection strategy include the collection object, collection metrics, collection frequency, filtering rules, and upload cycle. The node management sub-service layer is used to implement node registration, node authentication, and node monitoring. The permission management sub-service layer is used to allocate permissions to various user roles based on the RBAC model; The alarm management service layer is used to set alarm rules according to alarm type, scope of impact, and severity. The configuration management sub-service layer is used to configure global parameters and dedicated parameters for each functional module, and automatically generates a backup after each parameter configuration is successful.

6. The information probe acquisition system for resource pools as described in claim 2, characterized in that, The data collection and aggregation module adopts a distributed storage and hierarchical index architecture, which includes a data receiving layer, a data verification layer, a data storage layer, an index layer, and a query interface layer. The data receiving layer first writes the data uploaded by the management agent module into a distributed message queue for buffering, and then starts multiple threads to process the queue data in parallel, wherein each thread corresponds to a resource pool partition; The data verification layer performs integrity verification, consistency verification, and legality verification on the queue data according to the specifications defined by the data format management module, and temporarily stores the data that fails the verification to the abnormal data pool. The data storage layer uses multiple storage media to store verified data in layers according to the data access frequency and life cycle. It uses a Redis cluster to store real-time data, a MySQL or PostgreSQL cluster to store short-term data, and an HDFS distributed file system to store long-term data. All data is backed up off-site. The index layer filters and queries data according to basic dimensions, business dimensions, and technical dimensions; The query interface layer provides two types of standardized data query interfaces and connects to upper-layer applications.

7. The information probe acquisition system for resource pools as described in claim 2, characterized in that, The data acquisition strategy module includes a strategy modeling layer, a strategy optimization layer, and a strategy version management layer. The strategy modeling layer is used to provide preset strategy templates. Parameters can be modified based on the preset strategy templates through a visual drag-and-drop method, or the strategy parameters can be automatically configured according to the application conditions of the collection strategy. The strategy optimization layer is used to periodically analyze the execution effect of the acquisition strategy from the perspectives of acquisition efficiency and resource utilization, and generate optimization suggestions. The strategy version management layer is used to record the version number, creator, creation time and changes of each collection strategy, and supports version backtracking, version comparison and version archiving.

8. The information probe acquisition system for resource pools as described in claim 2, characterized in that, The data acquisition interface management module includes an interface definition layer, an interface authentication layer, an interface rate limiting layer, an interface monitoring layer, and an interface documentation management layer. The interface definition layer is used to uniformly define the basic information, request specifications, and response specifications of each data query interface; The interface authentication layer is used to verify the validity of the unique certificate held by the current module when calling the interface between internal modules; when calling the external data query interface, a temporary token is first obtained through username or password, and when calling the interface, it is determined whether the request header carries API key and token fields. If they do not exist, the authentication is considered to have failed. The interface rate limiting layer is used to set a rate limiting threshold for each interface. When the data call volume exceeds the rate limiting threshold, a retry is prompted and the retry time is given in the response header. When the interface call failure rate is ≥50%, the circuit breaker mechanism is automatically triggered, and new requests are rejected during the circuit breaker period. The interface monitoring layer is used to monitor the running status of the interface in real time, including call volume, success rate, response time and error code distribution. The interface document management layer is used to generate interface documents based on preset specifications. The document content includes interface details, debugging tools, and version history.

9. The information probe acquisition system for resource pools as described in claim 2, characterized in that, The data governance system constructs a full-process governance mechanism for identification, cleaning, standardization, enrichment, and quality assessment based on eight information characteristics: application name, version number, country of origin, application category, place of use, unit of use, user, and module of use. During the data collection phase, eight types of information features are initially automatically identified. For information features that cannot be automatically identified, they are marked as to be supplemented and reported to the management service module through the management agent module for display and supplementation. During the data cleaning phase, the identified information features are cleaned. During the data standardization phase, the cleaned information features are standardized in terms of field format uniformity, classification system uniformity, and coding rule uniformity, according to the specifications defined by the data format management module. During the data enrichment stage, multiple external data sources are connected to automatically supplement additional information on the standardized information features. For additional information that cannot be automatically supplemented, a manual enrichment and input interface is provided through the management service module. During the data quality assessment phase, a built-in data quality assessment model is used in the data collection and aggregation module to evaluate the quality of the enriched information features from four dimensions: completeness, accuracy, consistency, and timeliness. The quality levels are then classified, and corresponding improvement directions are provided.

10. The information probe acquisition system for resource pools as described in claim 2, characterized in that, It also includes a security strategy system; The security strategy system covers the entire chain of data transmission, data storage, data access, and data operation; During the data transmission phase, the data is encrypted across the entire link, and for sensitive data, a second encryption is performed using a dynamic encryption key. During the data storage phase, different encryption methods are adopted for structured data, unstructured data, and sensitive fields. Backup data is also stored in an off-site encrypted manner. When the data reaches the end of its storage life cycle, it is destroyed by logical overwrite or physical destruction, and a data destruction operation record is generated. During the data access phase, data access permissions are set based on user roles and user attributes, and access behavior is controlled. During the data manipulation phase, audit logs are generated for all critical operations and stored on the blockchain. Throughout the entire process, we regularly perform vulnerability scans and protection on all modules, and provide emergency response strategies in real time.

Images