Expandable container machine room data standardization processing method and system

Abstract

The present application relates to the technical field of data processing, in particular to a scalable container machine room data standardization processing method and system, comprising: S1, connecting different data sources, collecting and preprocessing original data; S2, filtering invalid original data, identifying newly added equipment of the container machine room, and generating metadata templates of the newly added equipment; S3, standardizing the filtered original data; S4, according to the dynamic sharding mechanism, allocating storage resources to the container machine room and storing the data in layers; wherein S3 comprises: according to the preset standardization processing rule, cleaning and converting the original data to generate converted data; according to the preset unified data model, structuring the converted data. The present application can standardize the data of different data sources and dynamically allocate storage resources, avoiding waste of container machine room resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a scalable container machine room data standardization processing method and system. BACKGROUND

[0002] As a modular data center, the container machine room is widely used in edge computing (5G base station, Internet of Things node), temporary business scenarios (disaster recovery, military deployment) and high-density computing demand (AI training cluster). Its data sources include sensors (temperature and humidity, power consumption), IT equipment logs, network traffic and business system data, with characteristics of multi-source heterogeneity, dynamic expansion, high timeliness, etc.

[0003] However, due to the diversity of data sources, the protocols of different manufacturers' equipment are different, such as Modbus, SNMP, MQTT, etc., and the unstructured log format is chaotic, and the number of connection nodes of each container machine room is different, and the data processing amount is also different. If the storage resources are allocated evenly, a large amount of resources will be wasted, and different data in a container machine room will also cause waste of storage resources because the importance of different data is different. If the data is stored in the same storage resource, it will cause data accumulation and affect data calling, and good storage resources will be occupied. SUMMARY

[0004] One of the purposes of the present application is to provide a scalable container machine room data standardization processing method, which can perform data standardization processing on data of different data sources and perform dynamic storage allocation to avoid waste of container machine room resources.

[0005] The basic scheme provided by the present application is as follows:

[0006] S1, connect different data sources to collect and preprocess raw data;

[0007] S2, filter invalid raw data, identify newly added equipment of the container machine room, and generate metadata templates of the newly added equipment; the raw data of the newly added equipment is generated according to the metadata templates, and the metadata templates include device data field definition, protocol conversion rule and storage adaptation identifier;

[0008] S3, standardize the filtered raw data;

[0009] S4, according to the dynamic sharding mechanism, allocate storage resources to the container machine room and store the data in layers; automatically adjust the storage shards according to the number of nodes of the container machine room, including:

[0010] Node state monitoring and service discovery are performed, Consul is used as a service discovery tool, the survival states of all nodes in the container room of the cluster are monitored in real time, each node registers with Consul when starting, and periodically sends a heartbeat packet, when a node is added or offline, Consul updates the node list through a health check mechanism; after a new node starts, it is automatically registered with Consul, triggering adjustment of the sharding strategy, Consul detects loss of node heartbeat and marks it as unavailable, triggering data migration;

[0011] A consistent hashing algorithm is used to map data shards to a virtual ring, and each node corresponds to multiple virtual nodes.

[0012] The number of virtual nodes is dynamically adjusted according to the hardware configuration of the node.

[0013] The S3 includes: according to a preset standardized processing rule, cleaning and converting the original data to generate converted data.

[0014] According to a preset unified data model, the converted data is structured.

[0015] Further, the S3 further includes: real-time monitoring of data flow, and offline completion of historical data.

[0016] Unstructured logs are automatically labeled and supplemented to the unified data model.

[0017] Further, in the S4, the triggering data migration conditions include: the number of nodes changes by more than a threshold, and the shard load is unbalanced.

[0018] The migration strategy includes: incremental migration, parallel migration, and double-write transition.

[0019] The migration process includes: shard locking, data synchronization, metadata updating, and traffic switching.

[0020] In the shard metadata management process, Consul Key-Value storage is used for metadata storage, recording shard distribution information.

[0021] Metadata synchronization listens to Consul Key-Value storage changes through Consul's Watch mechanism, and each node obtains the latest shard configuration in real time.

[0022] Further, the S4 includes: according to a preset data division rule, dividing data into hot data and cold data, and labeling, storing hot data in a time series database, and storing cold data in cold storage.

[0023] Further, the preset data division rule includes:

[0024] The visit frequency in a preset time period before the current time point is counted as the visit frequency, whether the visit frequency is greater than or equal to the predicted visit frequency threshold is judged, if yes, the data is determined as hot data, if not, the data is determined as cold data;

[0025] The heat of the data is calculated, heat=a*e^(-lambda*t), wherein a is an initial weight, lambda is an attenuation rate, and t is time, whether the heat is greater than or equal to a preset heat threshold is judged, if yes, the data is determined as hot data, if not, the data is determined as cold data;

[0026] If the visit frequency of the hot data of the time series database is less than a preset visit frequency threshold, or the heat of the hot data of the time series database is less than a preset heat threshold, the hot data of the time series database is migrated into the cold storage, and the label is changed;

[0027] If the visit frequency of the cold data of the cold storage is greater than or equal to a preset visit frequency threshold, or the heat of the hot data of the time series database is greater than or equal to a preset heat threshold, the cold data of the cold storage is migrated into the time series database, and the label is changed.

[0028] The second purpose of the application is to provide an expandable container machine room data standardization processing system.

[0029] The application provides a basic scheme two: an expandable container machine room data standardization processing system, which comprises:

[0030] A data acquisition module is used to connect different data sources, acquire original data and perform preprocessing.

[0031] A data preprocessing module is used to filter invalid original data, identify newly added equipment of the container machine room and generate a metadata template of the newly added equipment; original data of the newly added equipment is generated according to the metadata template, and the metadata template comprises device data field definition, protocol conversion rule and storage adaptation identifier.

[0032] A standardization processing module is used to perform standardization processing on the filtered original data.

[0033] An elastic storage module is used to allocate storage resources to the container machine room according to a dynamic sharding mechanism and perform hierarchical storage on data; the storage sharding is automatically adjusted according to the number of nodes of the container machine room.

[0034] The node state monitoring and service discovery are performed, Consul is used as a service discovery tool, the survival states of all nodes of the container room in the cluster are monitored in real time, each node is registered to Consul when starting, and a heartbeat packet is sent periodically, when a node is added or offline, Consul updates the node list through a health check mechanism; after a new node starts, the new node is automatically registered to Consul, triggering adjustment of a sharding strategy, Consul detects loss of a node heartbeat, marks the node as unavailable, and triggers data migration;

[0035] A consistent hashing algorithm is used to map data shards to a virtual ring, and each node corresponds to multiple virtual nodes.

[0036] The number of virtual nodes is dynamically adjusted according to the hardware configuration of the node.

[0037] The standardization processing module includes a standardization submodule and a structured submodule.

[0038] The standardization submodule is configured to clean and convert the original data according to a preset standardization processing rule to generate converted data.

[0039] The structured submodule is configured to structure the converted data according to a preset unified data model.

[0040] Further, the standardization processing module further includes a stream batch integrated processing submodule and an auxiliary labeling submodule.

[0041] The stream batch integrated processing submodule is configured to monitor data streams in real time and offline complete historical data.

[0042] The auxiliary labeling submodule is configured to automatically label unstructured logs and supplement the unified data model.

[0043] Further, in the elastic storage module, the triggering data migration conditions include that the number of nodes changes by more than a threshold value and shard load imbalance.

[0044] The migration strategy includes incremental migration, parallel migration, and double-write transition.

[0045] The migration process includes shard locking, data synchronization, metadata updating, and traffic switching.

[0046] In the shard metadata management process, Consul Key-Value storage is used for metadata storage to record shard distribution information.

[0047] Metadata synchronization listens to Consul Key-Value storage changes through the Watch mechanism of Consul, and each node obtains the latest shard configuration in real time.

[0048] Further, the elastic storage module divides data into hot data and cold data according to a preset data division rule, and performs marking, stores the hot data into a time series database, and stores the cold data into cold storage.

[0049] Further, the preset data division rule comprises:

[0050] The access frequency in a preset time period before the current time point is counted as the access frequency, and it is determined whether the access frequency is greater than or equal to a predicted access frequency threshold value, if yes, the data is determined as hot data, and if no, the data is determined as cold data;

[0051] The heat of the data is calculated, heat = a * e^(-lambda * t), wherein a is an initial weight, lambda is an attenuation rate, and t is time (days), and it is determined whether the heat is greater than or equal to a preset heat threshold value, if yes, the data is determined as hot data, and if no, the data is determined as cold data;

[0052] If the access frequency of the hot data of the time series database is less than a preset access frequency threshold value, or the heat of the hot data of the time series database is less than a preset heat threshold value, the hot data of the time series database is migrated into the cold storage, and the label is changed;

[0053] If the access frequency of the cold data of the cold storage is greater than or equal to a preset access frequency threshold value, or the heat of the hot data of the time series database is greater than or equal to a preset heat threshold value, the cold data of the cold storage is migrated into the time series database, and the label is changed.

[0054] Beneficial effects: In the scheme, the container machine room connects different data sources, collects and pre-processes original data, and filters out invalid original data to avoid wasting container machine room resources. The original data after filtering is standardized, which includes cleaning and converting the original data according to a preset standardization processing rule to generate converted data, and then using a preset unified data model to structure the converted data to ensure data consistency and complete data standardization processing of data from different data sources.

[0055] The storage resources of the container machine room are dynamically allocated according to a dynamic sharding mechanism, and the internal data is stored in layers to avoid unreasonable allocation of storage resources between container machine rooms and unreasonable allocation of internal data storage resources. The container machine room can allocate more storage resources to nodes or important container machine rooms, and the data in the container machine room can be stored in layers according to different preset standards to avoid data accumulation and affect data calling.

[0056] In summary, the scheme can standardize data from different data sources and dynamically allocate storage to avoid wasting container machine room resources. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating an embodiment of the scalable containerized data standardization processing method of the present invention.

[0058] Figure 2 This is a logic block diagram of Embodiment 2 of the scalable container room data standardization processing method of the present invention. Detailed Implementation

[0059] The following detailed description illustrates the specific implementation method:

[0060] Example 1

[0061] This embodiment is basically as shown in the appendix. Figure 1 As shown: The data standardization processing method for scalable containerized data centers includes the following:

[0062] S1. Connect to different data sources to collect and preprocess raw data.

[0063] Specifically, it connects to data sources with different protocols to obtain the raw data from the data sources.

[0064] Edge lightweight processors are deployed in each container room to perform raw data preprocessing. In this embodiment, the edge lightweight processors use an Agent developed based on Rust to perform local preprocessing and compression of raw data.

[0065] Local preprocessing includes, but is not limited to: removing null values, duplicate values, and outliers that exceed the corresponding preset range, and noise suppression; noise suppression uses a sliding window algorithm to smooth instantaneous fluctuations, such as instantaneous voltage spikes from power sensors.

[0066] Compression involves segmenting and aggregating high-frequency sensor data (such as temperature values ​​collected every second) into segments (e.g., taking the average value over 1 minute) to reduce the amount of data.

[0067] S2. Filter invalid raw data and identify new equipment in the container room, and generate metadata templates for the new equipment.

[0068] Specifically, invalid data, such as sensor drift values, are filtered according to preset dynamic rules.

[0069] The device fingerprint technology automatically identifies the type of new device and generates a metadata template; the original data is generated according to the metadata template.

[0070] Metadata templates provide unified adaptation rules for heterogeneous data generated by new devices, ensuring seamless integration of new device data into the overall data processing workflow and preventing data format chaos and improper storage allocation due to device addition. A metadata template consists of: device data field definitions (such as name, type, and unit), protocol conversion rules, and storage adaptation identifiers. After a new device starts and completes metadata template registration, it generates raw data according to the field format, data type, and protocol specifications agreed upon in the metadata template. If the device's native output format differs from the template, the data conversion is performed using the template's built-in adaptation rules, ultimately outputting raw data that conforms to the metadata template standard. This ensures that subsequent S3 standardization processing and S4 storage allocation can directly adapt to this data.

[0071] S3. Standardize the filtered raw data.

[0072] Specifically, S3 includes: cleaning and transforming the original data according to preset standardized processing rules to generate transformed data.

[0073] Based on a pre-defined unified data model, the transformed data undergoes structured processing to ensure data consistency.

[0074] The pre-defined unified data model in this embodiment is:

[0075] {

[0076] "metric": "temperature", / / Metric name

[0077] "timestamp": 1629984000, / / Timestamp (Unix format)

[0078] "value": 25.6, / / numerical value

[0079] "unit": "°C", / / unit

[0080] "location": "rack-A3", / / Device location

[0081] "tags": {"source": "sensor-123"} / / Extended tags (vendor, protocol version, etc.)

[0082] }

[0083] S3 also includes: real-time monitoring of data streams (e.g., temperature anomalies) and offline completion of historical data.

[0084] Automatically tag unstructured logs and supplement to the unified data model; in this embodiment, AI is used for auxiliary annotation, such as inputting the original log: "ERR: Disk sda1 write latency 1200ms exceeded threshold", and the AI outputs after annotation:

[0085] {

[0086] "log_level": "ERROR",

[0087] "device_type": "disk",

[0088] "error_code": "high_latency",

[0089] "metric": {"name": "write_latency", "value": 1200, "unit": "ms"}

[0090] }

[0091] Specifically, a pre-trained model such as BERT is used for text semantic understanding, and ONNX / TensorRT optimization is performed to run inference on the edge side.

[0092] All original data (whether sensors, logs or business systems) of the container machine room are labeled Quasi- normalized form storage, and can be directly used for monitoring, analysis or AI training.

[0093] S4, according to the dynamic sharding mechanism, the storage resources of the container machine room are allocated, and the data is stored in layers.

[0094] Specifically, it includes: automatically adjusting the storage sharding according to the number of nodes of the container machine room, including:

[0095] Node state monitoring and service discovery, Consul is used as a service discovery tool, and the survival state of all nodes of the container machine room in the cluster is monitored in real time. Each node registers with Consul when it starts, and sends a heartbeat packet regularly. When a node is added or offline, Consul updates the node list through the health check mechanism; after the new node starts, it is automatically registered to Consul, triggering the sharding strategy adjustment. Consul detects that the node heartbeat is lost (i.e. exceeds the timeout threshold, such as 30 seconds), and marks it as unavailable, triggering data migration.

[0096] Adopting the consistent hashing algorithm, the data shards are mapped to the virtual ring, and each node corresponds to multiple virtual nodes (virtual shards), ensuring that the increase and decrease of nodes only affect adjacent shards, reducing the amount of data migration.

[0097] The core goal of adopting the consistent hashing algorithm is to solve the problem of remapping a large amount of data when the node changes dynamically (newly added / offline) in the traditional hashing algorithm. By constructing a virtual ring with a value range of [0, 2^32-1], the node and the data are mapped to a specific location on the ring, and the data is matched to the nearest node in the clockwise direction for storage. Compared with the traditional hash, the advantage of consistent hash is that when the number of nodes changes, only the data in the adjacent interval of the affected node needs to be migrated, rather than all the data, greatly reducing the amount of data migration, adapting to the needs of dynamic expansion of container room nodes.

[0098] The specific implementation includes:

[0099] (1) Step 1: Constructing virtual ring and node mapping

[0100] Initialize the virtual ring: create an ordered array to store the hash values of all virtual nodes, and the array is sorted by hash value from small to large, simulating a virtual ring of [0, 2^32-1].

[0101] Physical node registration and virtual node generation: when the physical node (node-A) starts, register the node information (hardware configuration, IP, port) to Consul; calculate the number of virtual nodes N according to the hardware performance score of the node, generate N virtual node identifiers (node-A#1 to node-A#N); perform hash calculation on each virtual node identifier, virtual node hash value = SHA-1 (virtual node identifier) mod 2^32; store all virtual node hash values in the ordered array of the virtual ring, and record the "virtual node-physical node" mapping table, the mapping relationship is represented in Table 1.

[0102] Table 1

[0103]

[0104] Virtual ring sorting: sort the 32-bit hash values of all virtual nodes in ascending order to form an ordered virtual ring, and the example sorted result is: 10000 (node-A#1) → 18000 (node-B#1) → 25000 (node-A#2) → 32000 (node-B#2).

[0105] (2) Step 2: Data shard mapping to node

[0106] Data shard generation: The standardized dataset is divided into multiple data shards according to preset rules (such as time range, data type), and a unique shard key is generated for each shard (such as shard-20240520-08 for a time-based shard, indicating the data shard at 8 o'clock on May 20, 2024).

[0107] Data shard hash calculation: Perform a hash operation on the shard key, shard hash value = SHA-1(shard-key) mod 2^32, assuming the hash value of a certain shard shard-20240520-08 is 22000.

[0108] Find the target virtual node: In the ordered virtual ring, find the smallest virtual node hash value greater than the shard hash value through binary search algorithm, if not found, match the virtual node at the beginning of the ring (i.e. the circular characteristic of the ring structure). In the example, the smallest greater value corresponding to the shard hash value 22000 is 25000 (node-A#2).

[0109] Map to physical node: Determine the physical node corresponding to node-A#2 as node-A through the mapping relationship table, and finally store the data shard to node-A.

[0110] (Three) Step 3: Processing when node changes dynamically

[0111] Node addition (such as adding node-C):

[0112] Virtual node generation and ring update: Calculate the number of virtual nodes of node-C (assume 150), generate node-C#1 to node-C#150, and calculate their respective hash values (such as the hash value of node-C#50 is 20000); Insert all virtual node hash values of node-C into the ordered virtual ring, and the updated ring is sorted as: 10000 (node-A#1) → 18000 (node-B#1) → 20000 (node-C#50) → 25000 (node-A#2) → 32000 (node-B#2).

[0113] Determine the data migration range: The predecessor of the new virtual node 20000 (node-C#50) in the ring is 18000 (node-B#1), and the successor is 25000 (node-A#2); The data shards with hash values between 18000 and 20000 need to be migrated (originally stored in node-A corresponding to node-A#2), only the data in this interval is migrated, not all shards.

[0114] Data migration execution: According to the preset migration strategy, the above interval shards are migrated from node-A to node-C, and the shard-node mapping relationship in Consul is updated after the migration is completed.

[0115] Node offline (such as node-B failure):

[0116] Virtual node removal and ring update: Consul detects the loss of node-B heartbeat, marks node-B as unavailable, and removes all virtual nodes of node-B (18000, 32000) from the virtual ring; the updated ring order is: 10000 (node-A#1) → 20000 (node-C#50) → 25000 (node-A#2).

[0117] Determine the data migration range: The predecessor of the virtual node 18000 of the original node-B is 10000 (node-A#1), and the successor is 20000 (node-C#50), and the shards with hash values between 10000 and 18000 need to be migrated; the predecessor of the virtual node 32000 of the original node-B is 25000 (node-A#2), and the successor is 10000 (node-A#1, circular), and the shards with hash values between 25000 and 32000 need to be migrated.

[0118] Data migration execution: The shards of the above two intervals are migrated to node-C (10000~18000 interval, matching the successor node 20000) and node-A (25000~32000 interval, matching the successor node 10000) respectively, to ensure that data is not lost.

[0119] (Four) Step 4: Hash collision handling

[0120] When the hash values of different inputs (such as node-A#3 and node-C#10) are equal, the secondary hash method is used to solve the conflict: perform secondary hash calculation on the conflicting virtual node identifier: secondary hash value = SHA-256 (virtual node identifier) mod 2^32; if the secondary hash value still conflicts, continue to perform tertiary hash (SHA-512) until a unique hash value is obtained; store the final unique hash value in the virtual ring to ensure that the position of each virtual node in the ring is unique.

[0121] Through the above consistent hash algorithm, it is ensured that when the container room node is added or removed, the data migration amount is only related to the proportion of the number of virtual nodes of the added / removed node, which significantly reduces resource consumption and adapts to dynamic expansion requirements.

[0122] Adjust the number of virtual nodes dynamically according to the hardware configuration of the nodes (such as storage capacity, CPU performance), and high-performance nodes carry more shards.

[0123] Trigger data migration conditions, including: node number change (increase / decrease) exceeds threshold (such as ±10%), shard load imbalance (such as some node disk usage rate exceeds 80%).

[0124] Migration strategy, including: incremental migration, only migrate affected shard data, avoid full copy; parallel migration, pull data from multiple source nodes at the same time to improve migration speed; double-write transition, during migration, new data is written to new and old shards at the same time to ensure service continuity.

[0125] Migration process, including: shard locking, marking the shard to be migrated as "read-only" and suspending new write operations; data synchronization: copying shard data from old nodes to new nodes; metadata update: updating shard-node mapping in Consul; traffic switching: routing read-write requests to new nodes and unlocking old node shards.

[0126] In the process of shard metadata management, Consul Key-Value (KV) storage is used for metadata storage, recording shard distribution information, format example:

[0127] {

[0128] "shard-001": {

[0129] "nodes": ["node-A", "node-B"], / / primary and backup node

[0130] "range": "timestamp:[2023-01-01T00:00:00, 2023-01-31T23:59:59]", / / shard data range

[0131] "status": "active" / / shard status (active / migrating / read-only)

[0132] }

[0133] }

[0134] Metadata synchronization, listen to KV changes through Consul's Watch mechanism, each node real-time gets the latest shard configuration.

[0135] For example, a container room temperature sensor generates one data per second, and stores it by time range (such as one shard per hour), the adjustment process is as follows:

[0136] Initial state: 3 nodes (node-A, node-B, node-C), each node carries 10 time slice shards (00:00-23:00).

[0137] New node: node-D joins the cluster, Consul detects the new node and triggers shard rebalancing; the consistent hashing ring is recalculated, and part of the shards (such as 00:00-08:00) are migrated to node-D.

[0138] Data migration: node-A transfers 00:00-08:00 shard data to node-D in parallel, and during the migration, new data is written to node-A and node-D; after the migration is completed, Consul updates the shard mapping, and the traffic is switched to node-D.

[0139] Result: The load of each node decreases from 10 shards to 7-8 shards, and the resource utilization is more balanced.

[0140] When the node is expanded, the shard is automatically migrated to the new node without downtime; based on real-time load, the shard distribution is dynamically adjusted to avoid hot spot problems; the shard multi-copy mechanism (such as 3 copies) ensures data security, and single node failure has no effect; after the cold data shards are merged, they are archived to low-cost storage, saving resources. The dynamic shard mechanism monitors node changes through service discovery, calculates shard distribution through consistent hashing, migrates incremental data, and synchronizes metadata in real time in a closed loop process, achieving automatic adjustment of the storage shard strategy. This mechanism combines the agile service discovery of Consul and the strong consistency of Raft to ensure that when the container room node is dynamically expanded, it can still maintain high-performance and high-availability data storage services.

[0141] According to the preset data division rule, the data is divided into hot data and cold data, and is marked, and the hot data is stored into the time series database, and the cold data is stored into the cold storage; wherein the cold storage is archived to the object storage.

[0142] The preset data division rule includes:

[0143] The number of accesses in a preset time period before the current time point is counted as the access frequency, and it is judged whether the access frequency is greater than or equal to the predicted access number threshold, if yes, the data is determined as hot data, if not, the data is determined as cold data.

[0144] The heat of the data is calculated, heat = a*e^(-λ*t), wherein a is the initial weight, λ is the decay rate, t is the time (day), and it is judged whether the heat is greater than or equal to the preset heat threshold, if yes, the data is determined as hot data, if not, the data is determined as cold data.

[0145] If the access frequency of the hot data of the time series database is less than a preset access frequency threshold, or the hotness of the hot data of the time series database is less than a preset hotness threshold, the hot data of the time series database is migrated to the cold storage, and the label is changed.

[0146] If the access frequency of the cold data of the cold storage is greater than or equal to a preset access frequency threshold, or the hotness of the hot data of the time series database is greater than or equal to a preset hotness threshold, the cold data of the cold storage is migrated to the time series database, and the label is changed; thereby realizing dynamic storage adjustment of data, and in the embodiment, the storage adjustment of the hot data and the cold data is set to scan the data access record at a preset time point in a preset period, and migration is performed. The distinction between the hot data and the cold data is a dynamic and multi-dimensional decision-making process, which needs to be comprehensively determined in combination with the access frequency and the hotness, and through the label, the storage life cycle and the automatic migration, efficient hierarchical storage management is realized, and finally the data storage target of high performance, low cost and easy expansion is achieved.

[0147] Embodiment two

[0148] The embodiment is basically as shown in the accompanying drawings Figure 2 The expandable container room data standardization processing system includes the following contents:

[0149] A data acquisition module is configured to connect different data sources and perform original data acquisition and preprocessing.

[0150] Specifically, the data acquisition module includes a protocol adapter and an edge processor.

[0151] The protocol adapter is configured to connect data sources of different protocols and acquire original data of the data sources; the protocol adapter supports multi-protocol access, such as HTTP / MQTT / Modbus, and performs protocol extension through plug-in.

[0152] The edge lightweight processor is configured to be deployed in each container room and perform original data preprocessing; in the embodiment, the edge lightweight processor uses an Agent developed based on Rust to perform local preprocessing and compression of the original data.

[0153] The local preprocessing includes but is not limited to removing null values, duplicate values, abnormal values exceeding a corresponding preset range, and noise suppression; the noise suppression smoothes transient fluctuations through a sliding window algorithm, such as transient voltage peaks of a power sensor.

[0154] The compression is segmentation and aggregation (such as taking an average value in 1 minute) of high-frequency collected sensor data (such as temperature values collected every second), to reduce the data volume.

[0155] A data preprocessing module is configured to filter invalid original data, identify newly added equipment of the container room, and generate a metadata template of the newly added equipment.

[0156] Specifically, the data preprocessing module includes: a dynamic data cleaning submodule and a metadata registration submodule.

[0157] The dynamic data cleaning submodule is used to filter invalid data, such as sensor drift values, according to preset dynamic rules.

[0158] The metadata registration submodule is used to automatically identify new device types using device fingerprint technology and generate metadata templates; the original data is generated according to the metadata template.

[0159] The standardization module is used to standardize the filtered raw data.

[0160] Specifically, the standardization processing module includes: a standardization submodule and a structured submodule.

[0161] The standardization submodule is used to clean and transform the raw data according to preset standardization processing rules, and generate the transformed data.

[0162] The structured submodule is used to perform structured processing on the transformed data according to a preset unified data model, ensuring data consistency.

[0163] The pre-defined unified data model in this embodiment is:

[0164] {

[0165] "metric": "temperature", / / Metric name

[0166] "timestamp": 1629984000, / / Timestamp (Unix format)

[0167] "value": 25.6, / / numerical value

[0168] "unit": "°C", / / unit

[0169] "location": "rack-A3", / / Device location

[0170] "tags": {"source": "sensor-123"} / / Extended tags (vendor, protocol version, etc.)

[0171] }

[0172] In addition, the standardization processing module also includes: a batch processing submodule and an auxiliary annotation submodule;

[0173] The flow batch integrated processing submodule is used for real-time monitoring of data flow (such as temperature anomaly) and offline completion of historical data at the same time.

[0174] The auxiliary labeling submodule is used for automatically labeling unstructured logs and supplementing to the unified data model. In the embodiment, the auxiliary labeling submodule adopts AI for auxiliary labeling. For example, the AI labels the input original log "ERR: Disk sda1 write latency 1200ms exceeded threshold" and outputs:

[0175] {

[0176] "log_level": "ERROR",

[0177] "device_type": "disk",

[0178] "error_code": "high_latency",

[0179] "metric": {"name": "write_latency", "value": 1200, "unit": "ms"}

[0180] }

[0181] A pre-trained model such as BERT is adopted for text semantic understanding, and ONNX / TensorRT optimization is performed to run inference on the edge side.

[0182] All original data (whether sensors, logs or business systems) of the container machine room are stored in a standardized form and can be directly used for monitoring, analysis or AI training.

[0183] The elastic storage module is used for storage resource allocation of the container machine room according to a dynamic sharding mechanism, and layered storage of data.

[0184] Specifically, the elastic storage module automatically adjusts the storage sharding according to the number of nodes of the container machine room.

[0185] Node state monitoring and service discovery are performed, Consul is used as a service discovery tool, the survival state of all nodes of the container machine room in the cluster is monitored in real time, each node registers with Consul when starting and sends a heartbeat packet at regular intervals, when a node is added or goes offline, Consul updates the node list through a health check mechanism; after a new node starts, it is automatically registered with Consul, triggering adjustment of the sharding strategy, Consul detects loss of node heartbeat (i.e. exceeding a timeout threshold such as 30 seconds), marks it as unavailable, and triggers data migration.

[0186] Using consistent hashing algorithm, data shards are mapped to a virtual ring, and each node corresponds to multiple virtual nodes (virtual shards), ensuring that node addition or reduction only affects adjacent shards, reducing data migration volume.

[0187] The core goal of using consistent hashing algorithm is to solve the problem of large amount of data needing to be remapped when the node changes dynamically (addition / line). By constructing a virtual ring with a value range of [0, 2^32-1], the node and data are mapped to a specific location on the ring, and the data is matched to the nearest node in the clockwise direction for storage. Compared with traditional hashing, the advantage of consistent hashing is that when the number of nodes changes, only the data in the adjacent interval of the affected node needs to be migrated, rather than all the data, greatly reducing the data migration volume and adapting to the needs of dynamic expansion of container room nodes.

[0188] The specific implementation includes:

[0189] (1) Step 1: Constructing virtual ring and node mapping

[0190] Initialize the virtual ring: create an ordered array to store the hash values of all virtual nodes, the array is sorted by hash value from small to large, simulating a virtual ring of [0, 2^32-1].

[0191] Physical node registration and virtual node generation: when the physical node (node-A) starts, register the node information (hardware configuration, IP, port) to Consul; calculate the number of virtual nodes N according to the hardware performance score of the node, generate N virtual node identifiers (node-A#1 to node-A#N); perform hash calculation on each virtual node identifier, virtual node hash value = SHA-1 (virtual node identifier) mod 2^32; store all virtual node hash values in the ordered array of the virtual ring, and record the "virtual node-physical node" mapping table, the mapping relationship is represented as Table 2.

[0192] Table 2

[0193]

[0194] Virtual ring sorting: sort the 32-bit hash values of all virtual nodes in ascending order to form an ordered virtual ring, the example sorted result is: 10000 (node-A#1) → 18000 (node-B#1) → 25000 (node-A#2) → 32000 (node-B#2).

[0195] (2) Step 2: Data shard mapping to node

[0196] Data shard generation: The standardized dataset is divided into multiple data shards according to preset rules (such as time range, data type), and a unique shard-key is generated for each shard (such as shard-20240520-08 for a time-based shard, indicating the data shard at 8 o'clock on May 20, 2024).

[0197] Data shard hash calculation: Perform a hash operation on the shard-key, shard hash value = SHA-1(shard-key) mod 2^32, assuming the hash value of a certain shard shard-20240520-08 is 22000.

[0198] Find the target virtual node: In the ordered virtual ring, find the smallest virtual node hash value greater than the shard hash value through binary search algorithm, if not found, match the virtual node at the beginning of the ring (i.e. the circular characteristic of the ring structure). In the example, the smallest greater value corresponding to the shard hash value 22000 is 25000 (node-A#2).

[0199] Map to physical node: Determine the physical node corresponding to node-A#2 as node-A through the mapping relationship table, and finally store the data shard to node-A.

[0200] (Three) Step 3: Processing when node changes dynamically

[0201] Node addition (such as adding node-C):

[0202] Virtual node generation and ring update: Calculate the number of virtual nodes of node-C (assume 150), generate node-C#1 to node-C#150, and calculate their respective hash values (such as the hash value of node-C#50 is 20000); Insert all virtual node hash values of node-C into the ordered virtual ring, and the updated ring is sorted as: 10000 (node-A#1) → 18000 (node-B#1) → 20000 (node-C#50) → 25000 (node-A#2) → 32000 (node-B#2).

[0203] Determine the data migration range: The predecessor of the new virtual node 20000 (node-C#50) in the ring is 18000 (node-B#1), and the successor is 25000 (node-A#2); The data shards with hash values between 18000 and 20000 need to be migrated (originally stored in node-A corresponding to node-A#2), only the data in this interval is migrated, not all shards.

[0204] Data migration execution: According to the preset migration strategy, the above interval shards are migrated from node-A to node-C, and the shard-node mapping relationship in Consul is updated after the migration is completed.

[0205] Node offline (such as node-B failure):

[0206] Virtual node removal and ring update: Consul detects the loss of node-B heartbeat, marks node-B as unavailable, and removes all virtual nodes of node-B (18000, 32000) from the virtual ring; the updated ring order is: 10000 (node-A#1) → 20000 (node-C#50) → 25000 (node-A#2).

[0207] Determine the data migration range: The predecessor of the virtual node 18000 of the original node-B is 10000 (node-A#1), and the successor is 20000 (node-C#50), and the shards with hash values between 10000 and 18000 need to be migrated; the predecessor of the virtual node 32000 of the original node-B is 25000 (node-A#2), and the successor is 10000 (node-A#1, circular), and the shards with hash values between 25000 and 32000 need to be migrated.

[0208] Data migration execution: The shards of the above two intervals are migrated to node-C (10000~18000 interval, matching the successor node 20000) and node-A (25000~32000 interval, matching the successor node 10000) respectively, to ensure that data is not lost.

[0209] (Four) Step 4: Hash collision handling

[0210] When the hash values of different inputs (such as node-A#3 and node-C#10) are equal, the secondary hash method is used to solve the conflict: perform secondary hash calculation on the conflicting virtual node identifier: secondary hash value = SHA-256 (virtual node identifier) mod 2^32; if the secondary hash value still conflicts, continue to perform tertiary hash (SHA-512) until a unique hash value is obtained; store the final unique hash value in the virtual ring to ensure that the position of each virtual node in the ring is unique.

[0211] Through the above consistent hash algorithm, it is ensured that when the container room node is added or removed, the data migration amount is only related to the proportion of the number of virtual nodes of the added / removed node, which significantly reduces resource consumption and adapts to dynamic expansion requirements.

[0212] The number of virtual nodes is dynamically adjusted based on the node's hardware configuration (such as storage capacity and CPU performance), with high-performance nodes supporting more shards.

[0213] The conditions that trigger data migration include: changes in the number of nodes (increase / decrease) exceeding a threshold (e.g., ±10%), and uneven shard load (e.g., disk utilization of a certain node exceeds 80%).

[0214] Migration strategies include: incremental migration, which migrates only the affected shard data to avoid full replication; parallel migration, which pulls data from multiple source nodes simultaneously to improve migration speed; and dual-write transition, during which new data is written to both the old and new shards to ensure service continuity.

[0215] The migration process includes: shard locking, marking the shard to be migrated as "read-only" and pausing new write operations; data synchronization: copying shard data from the old node to the new node; metadata update, updating the shard-node mapping relationship in Consul; and traffic switching, routing read and write requests to the new node and unlocking the shards on the old node.

[0216] During the fragment metadata management process, Consul Key-Value (KV) storage is used to record fragment distribution information. Example format:

[0217] {

[0218] "shard-001": {

[0219] "nodes": ["node-A", "node-B"], / / Primary and backup replica nodes

[0220] "range": "timestamp:[2023-01-01T00:00:00, 2023-01-31T23:59:59]", / / Fragmented data range

[0221] "status": "active" / / Fragment status (active / migrating / read-only)

[0222] }

[0223] }

[0224] Metadata synchronization is achieved by monitoring key-value changes through Consul's Watch mechanism, allowing each node to obtain the latest sharding configuration in real time.

[0225] Taking the example of a temperature sensor in a container room generating one data point per second, and storing it in segments according to time ranges (e.g., one segment per hour), the adjustment process is as follows:

[0226] Initial state: 3 nodes (node-A, node-B, node-C), each node carries 10 time slice shards (00:00-23:00).

[0227] New node: node-D joins the cluster, Consul detects the new node and triggers shard rebalancing; the consistent hashing ring is recalculated, and part of the shards (such as 00:00-08:00) are migrated to node-D.

[0228] Data migration: node-A transfers 00:00-08:00 shard data to node-D in parallel, and during the migration, new data is written to node-A and node-D; after the migration is completed, Consul updates the shard mapping, and the traffic is switched to node-D.

[0229] Result: The load of each node decreases from 10 shards to 7-8 shards, and the resource utilization is more balanced.

[0230] When the node is expanded, the shard is automatically migrated to the new node without downtime; based on real-time load, the shard distribution is dynamically adjusted to avoid hot spot problems; the shard multi-copy mechanism (such as 3 copies) ensures data security, and single node failure has no effect; after the cold data shards are merged, they are archived to low-cost storage, saving resources. The dynamic shard mechanism monitors node changes through service discovery, calculates shard distribution based on consistent hashing, migrates incremental data, and synchronizes metadata in real time to form a closed loop process, which realizes the automatic adjustment of the storage shard strategy. This mechanism combines the agile service discovery of Consul and the strong consistency of Raft to ensure that when the container room node is dynamically expanded, the system can still maintain high-performance and high-availability data storage services.

[0231] Elastic storage module, according to the preset data division rule, divide the data into hot data and cold data, and mark, store the hot data into the time series database, and store the cold data into the cold storage.

[0232] Among them, the preset data division rule includes:

[0233] Count the number of accesses in the preset time period before the current time point as the access frequency, and determine whether the access frequency is greater than or equal to the predicted access frequency threshold, if yes, the data is determined as hot data, if not, the data is determined as cold data.

[0234] Calculate the heat of the data, heat = a*e^(-λ*t), where a is the initial weight, λ is the decay rate, t is the time (days), and determine whether the heat is greater than or equal to the preset heat threshold, if yes, the data is determined as hot data, if not, the data is determined as cold data.

[0235] If the access frequency of the hot data of the time series database is less than the preset access frequency threshold, or the hotness of the hot data of the time series database is less than the preset hotness threshold, the hot data of the time series database is migrated to the cold storage, and the label is changed.

[0236] If the access frequency of the cold data of the cold storage is greater than or equal to the preset access frequency threshold, or the hotness of the hot data of the time series database is greater than or equal to the preset hotness threshold, the cold data of the cold storage is migrated to the time series database, and the label is changed; thereby realizing dynamic storage adjustment of data, and in the embodiment, the storage adjustment of the hot data and the cold data is set to scan the data access record at a preset time point in a preset period, and migration is performed. The distinction between the hot data and the cold data is a dynamic and multi-dimensional decision-making process, which needs to be comprehensively determined in combination with the access frequency and the hotness, and through the label, the storage life cycle and the automatic migration, efficient hierarchical storage management is realized, and finally the data storage goals of high performance, low cost and easy expansion are achieved.

[0237] The above is only an embodiment of the present application, and common knowledge of specific structures and properties in the scheme is not described in detail, and the ordinary skilled person in the art knows all the ordinary technical knowledge in the field of the application before the application date or the priority date, can know all the prior art in the field, and has the ability to apply conventional experimental means before the date, the ordinary skilled person in the art can improve and implement the present scheme under the guidance of the present application, and some typical known structures or known methods should not be an obstacle for the ordinary skilled person in the art to implement the present application. It should be noted that for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should also be considered as the protection scope of the present application, which will not affect the effect and practicality of the patent. The protection scope of the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.

Claims

1. Scalable container machine room data standardization processing method, characterized in that, Comprise the following contents: S1, connect different data sources, collect and preprocess raw data; S2, filter invalid raw data, identify new equipment in container machine room, generate metadata template of new equipment; S3, standardize the filtered raw data; including: according to the preset standardization processing rule, the raw data is cleaned and converted to generate the converted data; according to the preset unified data model, the converted data is structured; real-time monitoring of data flow, while offline completion of historical data; automatically label unstructured log and supplement to the unified data model; S4, according to the dynamic sharding mechanism, the storage resources of the container machine room are allocated, and the data is stored in layers; according to the number of nodes of the container machine room, the storage sharding is automatically adjusted, including: Node state monitoring and service discovery, Consul is used as the service discovery tool, the survival state of all nodes in the cluster is monitored in real time, each node registers with Consul when starting, and sends heartbeat package regularly, when the node is added or offline, Consul updates the node list through the health check mechanism; after the new node starts, it is automatically registered to Consul, triggering the sharding strategy adjustment, Consul detects the loss of node heartbeat, marks it as unavailable, and triggers data migration; Using consistent hashing algorithm, the data sharding is mapped to virtual ring, each node corresponds to multiple virtual nodes; According to the hardware configuration of the node, the number of virtual nodes is dynamically adjusted; According to the preset data division rule, the data is divided into hot data and cold data, and the hot data is stored in the time series database, and the cold data is stored in the cold storage; The preset data division rule includes: counting the access frequency in the preset time period before the current time point as the access frequency, judging whether the access frequency is greater than or equal to the predicted access frequency threshold, if yes, the data is determined as hot data, if not, the data is determined as cold data; calculating a hotness of the data, hotness = a e^(-λ t), where a is an initial weight, λ is a decay rate, and t is time, determining whether the hotness is greater than or equal to a preset hotness threshold, and if so, determining that the data is hot data, and if not, determining that the data is cold data; If the access frequency of the hot data of the time series database is less than the preset access frequency threshold, or the hotness of the hot data of the time series database is less than the preset hotness threshold, the hot data of the time series database is migrated to the cold storage, and the label is changed; If the access frequency of the cold data of the cold storage is greater than or equal to the preset access frequency threshold, or the hotness of the hot data of the time series database is greater than or equal to the preset hotness threshold, the cold data of the cold storage is migrated to the time series database, and the label is changed.

2. The scalable containerized machine room data standardization method of claim 1, wherein, In S4, the trigger data migration conditions include: the number of nodes changes more than the threshold, the sharding load is unbalanced; Migration strategy, including: incremental migration, parallel migration, double write transition; Migration process, including: sharding locking, data synchronization, metadata updating, traffic switching; In the process of sharding metadata management, Consul Key-Value storage is used to record sharding distribution information Metadata synchronization, Consul Key-Value storage changes are monitored through Consul Watch mechanism, and each node obtains the latest sharding configuration in real time.

3. The scalable container machine room data standardization processing system, characterized in that, Data acquisition module, for connecting different data sources, collecting and preprocessing raw data; The data preprocessing module is used for filtering invalid raw data and identifying newly added equipment of the container machine room to generate a metadata template of the newly added equipment. The standardized processing module is used for standardizing the filtered raw data, including a standardization submodule, a structuring submodule, a flow-batch integrated processing submodule, and an auxiliary labeling submodule. The standardization submodule is used for cleaning and converting the raw data according to a preset standardization processing rule to generate converted data. The structuring submodule is used for structuring the converted data according to a preset unified data model. The flow-batch integrated processing submodule is used for monitoring a data stream in real time and simultaneously offline completing historical data. The auxiliary labeling submodule is used for automatically labeling unstructured logs and supplementing the unified data model. The elastic storage module is used for allocating storage resources to the container machine room according to a dynamic sharding mechanism and storing data in layers, and automatically adjusting storage shards according to the number of nodes of the container machine room. Node state monitoring and service discovery are performed, Consul is used as a service discovery tool to monitor the survival state of all nodes of the container machine room in the cluster in real time, each node registers with Consul when starting and periodically sends a heartbeat packet, and Consul updates a node list through a health check mechanism when a node is added or goes offline; a new node is automatically registered with Consul after starting, triggering shard strategy adjustment, Consul detects loss of a node heartbeat and marks the node as unavailable, triggering data migration. A consistent hashing algorithm is used to map data shards to a virtual ring, and each node corresponds to multiple virtual nodes. The number of virtual nodes is dynamically adjusted according to the hardware configuration of the node. According to a preset data division rule, data is divided into hot data and cold data, and is labeled, hot data is stored in a time series database, and cold data is stored in cold storage. The preset data division rule includes: The number of accesses in a preset time period before the current time point is counted as an access frequency, and it is determined whether the access frequency is greater than or equal to a predicted access frequency threshold, if yes, the data is determined as hot data, and if no, the data is determined as cold data; calculating a hotness of the data, hotness = a e^(-λ t), where a is an initial weight, λ is a decay rate, and t is time, determining whether the hotness is greater than or equal to a preset hotness threshold, and if so, determining that the data is hot data, and if not, determining that the data is cold data; If the access frequency of the hot data of the time series database is less than a preset access frequency threshold, or the hotness of the hot data of the time series database is less than a preset hotness threshold, the hot data of the time series database is migrated to the cold storage and the label is changed; If the access frequency of the cold data of the cold storage is greater than or equal to a preset access frequency threshold, or the hotness of the hot data of the time series database is greater than or equal to a preset hotness threshold, the cold data of the cold storage is migrated to the time series database and the label is changed.

4. The expandable container cabin data standardization processing system according to claim 3, characterized in that, In the elastic storage module, the triggering data migration conditions include a change in the number of nodes exceeding a threshold and unbalanced shard load. The migration strategy includes incremental migration, parallel migration, and double-write transition. The migration process includes shard locking, data synchronization, metadata updating, and traffic switching. In the shard metadata management process, metadata storage uses Consul Key-Value storage to record shard distribution information. Metadata synchronization, through Consul's Watch mechanism to listen to Consul Key-Value storage changes, each node to obtain the latest real-time configuration of the shard.

Images