A Doris method for storing device data based on industrial Internet

Abstract

The present invention relates to the field of data storage technology, specifically disclosing a Doris method for device data storage based on the Industrial Internet. The method comprises the following specific steps: S1: acquiring industrial device node data; S2: preprocessing the acquired data; S3: uploading the preprocessed data to Kafka; S4: Doris consuming and storing Kafka data; S5: monitoring and performance optimization; and S6: expansion: secondary utilization and downstream analysis. The method significantly reduces data processing latency, improving real-time performance; enhances data quality and consistency, reducing missing and anomalies; and automates partitioning and storage cleanup to reduce operation and maintenance costs and improve scalability.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a Doris method for device data storage based on the Industrial Internet. Background Art

[0002] With the development of the Industrial Internet and the Internet of Things, the amount of equipment data collected has surged, and the requirements for real-time performance, reliability, and horizontal expansion are increasing.

[0003] Existing solutions primarily include: writing data directly into relational databases, big data storage based on the Hadoop ecosystem, message middleware + database integration, and dedicated time series databases. However, these solutions have significant shortcomings: relational databases are prone to performance bottlenecks under high concurrent writes, requiring database and table sharding or capacity expansion, resulting in cost and complex operations and maintenance. Big data solutions require the deployment of multiple clusters, including Kafka, HDFS, YARN, Hive / Spark, resulting in complex architectures and high operations and maintenance costs. HDFS's multiple replica storage wastes resources and suffers from poor real-time performance. Message middleware + database solutions require additional consumer programs, resulting in complex consumer-side logic and limited scalability. Historical data archiving is inconsistent and prone to errors. Dedicated time series databases, while optimized for time series data, are prone to cluster performance bottlenecks under massive concurrent writes, and their TTL retention policies lack flexibility, making them unable to meet the differentiated needs of different devices and business scenarios. Furthermore, time series databases can also experience problems such as data skew and query timeouts, increasing the risk of system instability.

[0004] In general, existing technologies cannot simultaneously take into account high-concurrency writing, second-level analysis, controllable storage costs and flexible partition management. Therefore, a new device data acquisition data storage method is urgently needed to solve the above-mentioned defects. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems of writing performance bottleneck, complex system architecture, high storage cost and inflexible partition management in the equipment data acquisition and data storage in the prior art, and to propose a Doris method for equipment data storage based on the Industrial Internet to achieve high-concurrency writing, real-time query and automated data archiving management.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A Doris method for storing device data based on the Industrial Internet includes the following specific steps:

[0008] S1: Acquire industrial equipment node data: collect data from on-site PLC or gateway devices in real time to prepare for subsequent cleaning and storage;

[0009] S2: Preprocess the acquired data: remove duplicates, complete, remove noise, and standardize the original collected data to ensure the quality and consistency of the data subsequently stored;

[0010] S3: Uploads pre-processed data to Kafka. This uses distributed message queues to achieve data buffering and decoupling, enabling parallel downstream consumption and good scalability.

[0011] S4: Doris consumes and stores Kafka data: Doris pulls JSON data from Kafka in real time and writes it to a relational table. It also uses dynamic partitioning to automate data partitioning and expiration cleanup.

[0012] S5: Monitoring and performance optimization: Monitor the end-to-end link from data collection to storage, promptly identify bottlenecks and anomalies, and perform targeted optimization.

[0013] S6: Extension: Secondary utilization and downstream analysis, while ensuring data quality and real-time performance, support subsequent data warehouse docking, real-time computing and machine learning applications.

[0014] As a further technical solution of the present invention, S1 specifically includes:

[0015] S11: Collection method selection: PLC direct connection collection or gateway forwarding collection;

[0016] S111: PLC direct connection acquisition: The on-site PLC establishes a communication protocol such as Modbus (TCP / RTU), PROFINET, or EtherNet / IP with the host computer via Ethernet or serial port. The system initiates a read request to the PLC at a preset sampling frequency (for example, once per second or once every ten seconds) to obtain the real-time value of a specified register or data block. It is important to confirm the register range (start and end addresses) read during each acquisition. This is configured by the equipment manufacturer or project implementer based on actual needs.

[0017] S112: Gateway forwarding and collection: If some field devices do not support traditional PLC protocols, an industrial edge gateway (such as an OPC UA gateway, Modbus gateway, Siemens S7 gateway, etc.) is required to collect and parse the data of the device's own protocol. The parsed raw information (usually key-value pairs or a custom binary format) is then converted into a unified data format (such as JSON) and forwarded to the back-end system.

[0018] S12: Transmission protocol and upload method: MQTT upload or HTTP interface upload;

[0019] S13: Timestamp and Unique Identifier Processing: Both the data collection end and the gateway must use a consistent time base, typically millisecond-level Unix timestamps, which represent the number of milliseconds from January 1, 1970, to the current time. This ensures consistent time sequencing for data uploaded by different devices. For device unique identification, if the device itself does not have a globally unique ID, a fixed prefix (such as the gateway number) can be concatenated with the device's local number on the gateway. For example, "GW01_PLC05" can serve as the device's globally unique ID. This ensures that devices are not confused during subsequent processing and storage.

[0020] As a further technical solution of the present invention, the S12 specifically includes:

[0021] S121: MQTT upload: Using the lightweight MQTT protocol, the gateway or acquisition end publishes the processed data in the form of messages to the specified topic. Topics can be hierarchically organized according to "company number / factory number / workshop number / equipment number", for example: / CompanyA / Factory1 / Line2 / PLC_001; QoS (Quality of Service) is usually available at two levels: "at least once" or "exactly once". "At least once" ensures that messages will not be lost, but duplication may occur; "exactly once" consumes more resources; the message retention flag (retain) is generally set to "do not retain" to prevent existing subscribers from receiving outdated historical messages; information carried in the report includes device number, timestamp, and sensor readings; if the connection is disconnected or the server returns an error, the number of retries and retry interval can be set, and after multiple failed attempts, an alarm will be issued or the data will be written to a local backup to prevent data loss;

[0022] S122: HTTP interface upload: Send the collected data to the backend service through HTTP POST request; interface address example: http: / / <backend domain name>:<port> / api / v1 /

[0023] device / Data; specify Content-Type: application / in the HTTP request header

[0024] json, the request body is in standard JSON format. If the request fails due to network problems or backend exceptions, the client will try again according to the pre-set retry strategy (for example, a maximum of three retries, each with a half-second interval). If all attempts fail, the error information is recorded in the local log and an alarm is triggered to notify the operation and maintenance personnel.

[0025] As a further technical solution of the present invention, S2 specifically includes:

[0026] S21: Duplicate data filtering and discarding invalid data;

[0027] S211: Duplicate data filtering: If the same device uploads the same timestamp and data value to the backend multiple times within a short period of time, it is considered duplicate data. After receiving the data, the system first checks whether the device number and timestamp are exactly the same as the most recent one. If they are consistent, the later data is directly discarded to avoid redundant storage.

[0028] S212: Invalid Data Discard: Messages uploaded by devices often contain a "status code" field to indicate whether the sampling status is normal. If the status code is not "normal," the data is invalid and should be discarded without further processing. Furthermore, for key physical quantities such as temperature and pressure, if the values ​​significantly exceed the reasonable range defined in the product manual or by the business (for example, if the temperature exceeds the rated upper limit or falls below the rated lower limit), these values ​​are also considered abnormal data and are discarded and recorded in the log.

[0029] S22: Data Missing Filling and Interpolation: To ensure the continuity of time series data, it is sometimes necessary to fill in the missing moments during the collection process. A common practice is to check the time span between two adjacent valid samples of the same device. If the time interval does not exceed the preset maximum interval (for example, within 5 seconds), the linear relationship between the two previous and subsequent valid values ​​is used to estimate and fill in the missing points. If the interval is too large, no interpolation is performed and the previous valid value or null value is directly filled in to avoid error accumulation. For simple cases, "nearest value filling" is used to directly replace the valid sample value closest to the missing moment. If higher accuracy is required, advanced interpolation algorithms (such as polynomial interpolation and spline interpolation) can also be used at the application layer. However, the implementation cost and computational overhead are relatively high, and they are rarely used in general industrial IoT scenarios.

[0030] S23: Data cleaning and standardization: including unit unification, noise filtering, and field renaming and supplementation;

[0031] S231: Unit Unification: All physical quantities must be converted to standard units. For example, if the temperature unit uploaded by a field device is in Fahrenheit, the system should convert it to Celsius before proceeding to the next step to ensure consistency in subsequent analysis and storage. The same approach applies to various physical quantities such as pressure, humidity, current, and voltage.

[0032] S232: Noise filtering: Due to noise interference during field acquisition, data smoothing is sometimes required. Common methods include sliding average or median filtering. For example, averaging the values ​​of the most recent N samples can eliminate short-term pulse interference, or taking the median of N samples to further remove the influence of extreme values.

[0033] S233: Field renaming and supplementation: If the downstream system (such as Kafka producers) requires fixed field names and formats, the original field names must be uniformly renamed. For example, the "temp" field can be changed to "temperature" after cleaning; "press" can be changed to "pressure". If the message lacks certain predefined fields, default values ​​can be added (such as a numeric value set to 0 or a string value set to "unknown") or set to null, depending on business needs.

[0034] S24: Data encapsulation into JSON: After completing the above cleaning, the data is organized into a unified JSON structure, which usually contains the following parts: device_id: unique device identifier;

[0035] timestamp: millisecond-level Unix timestamp when the server receives the data; data: key-value pairs of core sensor readings (such as temperature, pressure, humidity, etc.); status: device status code, used to indicate whether the sampling is normal; metadata: optional information, such as gateway number, signal strength, collection end IP, and other auxiliary fields;

[0036] Example JSON (illustrative only):

[0037] { "device_id": "GW01_PLC05", "timestamp": 171XXXXXXX000,

[0038] "data": { "temperature": 42.8, "pressure": 1.05, "humidity": 58.2}, "status": 0, "metadata": { "gateway_id": "GW01", "signal_strength": -65}};

[0039] After adopting this fixed format, when sending to Kafka or calling the HTTP interface, the JSON structured data can be directly used as the message body to keep the meaning of each field consistent.

[0040] As a further technical solution of the present invention, S3 specifically includes:

[0041] S31: Kafka cluster and topic planning: Build at least three Kafka Brokers on the backend to ensure high system availability; design topic names according to business logic, such as "companyA_

[0042] "factory1_line2_equipment_data" aggregates data from the same production line or the same type of equipment under the same topic for centralized processing. Based on the peak message volume and system concurrency, configure an appropriate number of partitions for this topic. More partitions increase downstream parallel consumption capabilities, but also increase broker management overhead. Typically, the system's peak message rate (e.g., 100,000 messages per second) is evaluated, and then the maximum throughput of each partition is considered to determine an appropriate number of partitions, such as 8, 16, or 32. Set the replication factor to 3, meaning each message will have a replica on three brokers to ensure data readability even if any two nodes fail.

[0043] S32: Kafka Producer configuration;

[0044] S33: Message sending and exception handling: The Producer calls the asynchronous sending interface and monitors the sending result in the callback function: if the sending is successful, it will record the success log or ignore it; if the sending fails, the reason for the failure (such as network timeout, Broker unavailability, etc.) will be recorded in the log, and another attempt will be made according to the retry strategy; if it still fails after multiple retries, the message will be placed in the local temporary buffer queue and delivery will be attempted again after background recovery or manual intervention;

[0045] Throughput optimization: You can set batch sending parameters appropriately. For example, when a certain number of messages (such as 500) or a certain number of bytes (such as 32KB) are accumulated, they are sent all at once to avoid the network overhead caused by frequent small batch sending. At the same time, you can set "linger.ms" to tens of milliseconds to allow the Producer to package and send messages that arrive within a short period of time.

[0046] As a further technical solution of the present invention, the S32 specifically includes:

[0047] S321: Broker list: write the addresses of all Brokers into the Producer configuration, for example broker1:9092,broker2:9092,broker3:9092;

[0048] S322: Message Acknowledgement Mechanism (acks): It is recommended to set it to "all", which means that the message is sent successfully only after it has been written to all In-Sync replicas, ensuring that no data is lost.

[0049] S323: Retry times: Automatically retry when sending fails, for example, set to 3 times; if it still fails after multiple retries, an alarm will be triggered or written to the local log to prevent silent data loss;

[0050] S324: Serializer: Use string serialization when sending the device number as a key, and also use UTF-8 encoding serialization when sending a JSON string as a value;

[0051] S325: Partition allocation strategy: Use the default hash partition strategy, that is, use the device number to hash and then modulo the total number of partitions. This ensures that all data on the same device falls into the same partition, ensuring the timeliness and sequential consumption of data on the same device.

[0052] As a further technical solution of the present invention, the S4 specifically includes:

[0053] S41: Doris table structure and dynamic partition design: Create a wide table in Doris, the fields include: device_id, event_time, temperature, pressure, humidity, status, gateway_id, etc.; the table is partitioned by date range (Range Partitioning) and enable dynamic partitioning; the system automatically creates five partitions based on the current date, from "3 days before the current day" to "1 day after the current day", with partition naming format pYYYYMMDD. For example, p20250606 represents the partition for June 6, 2025. This ensures that data for that day and recent historical days has corresponding partitions, making queries more efficient; set a partition retention period, such as 30 days. That is, when a partition is created for a certain number of days (such as 30 days), the system will automatically delete it, realizing automatic cleanup of expired data, which makes storage space management more flexible; at the same time, the table is bucketed (Hash bucketing), for example, hash bucketing is performed by device_id, with 16 buckets set. This allows data from different devices or partitions to be distributed to multiple nodes in parallel during queries and writes, improving performance;

[0054] S42: Create a Routine Load task: Execute a special SQL statement on Doris to create a Routine Load task, specifying which Kafka topic the task pulls data from, the concurrency rate to use, the maximum number of records or bytes to pull per batch, and how to parse JSON. The mapping of JSON fields to table fields is provided by a JSONPaths file. The corresponding relationship is roughly as follows: $.device_id corresponds to the device_id in the table; $.timestamp corresponds to event_time, and the timestamp needs to be converted to the Doris DATETIME type when writing; $.data.temperature, $.data.pressure, and $.data.humidity correspond to the temperature, pressure, and humidity fields in the table, respectively; $.status corresponds to the status in the table;

[0055] $.metadata.gateway_id corresponds to the gateway_id in the table. In the Routine Load task configuration, you can specify the number of concurrent consumers (for example, 4 concurrent threads), the maximum number of rows per batch (for example, 5000) or the maximum byte size (for example, 10MB), and the strict mode switch. When strict mode is enabled, if a field in a JSON entry is missing or the type does not match, it will be considered "dirty data" and discarded. Logs will also be recorded to ensure data consistency.

[0056] S43: Data loading process and dirty data processing: Doris Routine Load pulls messages from Kafka partitions in parallel according to the specified concurrency. Each message that meets the threshold (number of rows or bytes) is stored in a temporary file. The system decompresses and parses the temporary file, extracts fields from the JSON according to the JSONPaths mapping, and writes them to the memory table. The memory table data is automatically allocated to the corresponding partition according to the partitioning strategy, and then the backend node persists the data in batches to the column store. For records that fail to parse, have mismatched field types, or are missing required fields, Routine Load will place the record in the "dirty data" queue and decide whether to ignore or issue an alarm based on the "maximum dirty data ratio" threshold. If the dirty data ratio of a batch exceeds the threshold (for example, 5%), a system alarm will be triggered, and manual inspection will be conducted to check whether the JSON format or JSONPaths mapping is incorrect.

[0057] S44: Dynamic partitioning and expiration cleanup: Since the dynamic partitioning function is turned on, Doris will automatically determine the current date every day and create 5 partitions from "3 days before the current day" to "1 day after the current day". For example, if today is June 6, 2025, p20250603, p20250604, p20250605, p20250606, and p20250607 will be created; when a partition is created for several days (such as 30 days) (for example, after p20250506 expires), the system will automatically delete the partition without human intervention, thereby saving storage space; if the production environment needs to retain historical data for a longer period of time, just change the "partition expiration time" parameter from 30 days to 60 days, 90 days or longer.

[0058] As a further technical solution of the present invention, the S5 specifically includes:

[0059] S51: End-to-end monitoring indicators: including collection end monitoring, Kafka monitoring and Doris monitoring;

[0060] S511: Acquisition end monitoring:

[0061] Is the sampling frequency stable? Regularly count the number of samples per second or per minute. If the sampling frequency drops sharply or is interrupted, check the PLC connectivity or gateway status.

[0062] Network packet loss rate and retry times: When the gateway delivers to the backend, if the network condition is poor, retries or packet loss may occur. Monitor the ratio of retries to total transmission times and issue an alarm when it exceeds a certain threshold (such as 5%).

[0063] S512: Kafka Monitoring:

[0064] Consumer Lag: This measures the difference between the consumption speed of downstream consumers (Routine Load) and the sending speed of producers (Producers). Specifically, it checks the difference between the "Current Write Offset" and the "Committed Offset" of each partition. When the lag value increases and continuously exceeds a certain threshold (such as 10,000 records), it indicates that downstream consumption is insufficient and needs to be expanded or the number of concurrent users adjusted.

[0065] Throughput: Count the message writing rate (messages / second) and byte rate of the Producer to Kafka to confirm whether the maximum carrying capacity of the cluster has been reached;

[0066] Partition distribution: Check whether messages are evenly distributed across partitions. If the load on a partition is much higher than that on other partitions, consider increasing the number of partitions or using a custom partitioning strategy.

[0067] S513: Doris monitoring:

[0068] Routine Load Delay: This measures the time difference between the latest data in Kafka being written to Doris. You can regularly query the maximum event time (event_time) of the inserted data in the table and compare it with the current system time. If the delay lasts for more than 5 minutes, there may be a downstream write performance bottleneck, which requires troubleshooting.

[0069] Write QPS / TPS: Counts the number of queries per second or transactions per second for each BE node during write operations. If a node is continuously under high load (e.g., CPU usage exceeds 80%), consider capacity expansion or parameter optimization.

[0070] Partition size: Monitor the data volume of each partition. If the data volume of a partition far exceeds expectations (for example, exceeding 100GB), consider adjusting the number of buckets or further refining the historical partitions.

[0071] S52: Performance optimization strategy: including Kafka optimization and Doris optimization.

[0072] As a further technical solution of the present invention, the S52 specifically includes:

[0073] S521: Kafka optimization:

[0074] Batch sending and compression: Set an appropriate batch waiting time (such as tens of milliseconds) on the Producer side, aggregate multiple messages arriving within a short period of time and send them together. Enable compression algorithms such as Snappy or LZ4 to reduce network bandwidth usage and increase the Broker's write speed.

[0075] Properly configure the number of concurrent consumers: If the number of concurrent consumers in Routine Load is too low, the consumption speed will not keep up with the production speed; if the number of concurrent consumers is too high, Doris's write tasks will compete for resources on the backend nodes. It is necessary to find the optimal balance of concurrent consumers based on the CPU, memory, and disk I / O indicators of the BE node;

[0076] S522: Doris optimization:

[0077] Column encoding and compression: For commonly used fields in the table (such as temperature, pressure, and other numerical columns), column compression encoding methods (such as RLE and dictionary compression) are used to significantly reduce storage usage and improve scanning efficiency.

[0078] Small file merging: If frequent writing results in a large number of small files, which will affect query speed, you can perform partition file merging operations regularly to automatically merge small files into large files.

[0079] Statistics maintenance: Regularly perform analysis operations on tables to update column histogram statistics, allowing the query optimizer to select a better execution plan.

[0080] As a further technical solution of the present invention, the S6 specifically includes:

[0081] S61: Data warehouse connection: In Doris, wide tables are constructed into fact tables, and then dimension tables are designed according to business needs. If more complex online analytical processing (OLAP) is required, data can be synchronized to downstream data warehouses (such as Hive / Hadoop, ClickHouse, Presto, etc.) through ETL or scheduled tasks for offline reporting and multidimensional analysis. In the same Doris cluster, views or hierarchical table structures can also be combined to provide data of different granularities to BI tools in a timely manner.

[0082] S62: Real-time computing and alerting: Using Apache Flink and Spark Streaming's streaming processing engines between Kafka and Dori, or in parallel with Doris, pre-processed data is directly consumed for complex event processing (such as anomaly detection and trend analysis). When indicators such as temperature and pressure exceed preset thresholds, SMS or email alerts are immediately triggered. Based on the CEP (complex event processing) model, multiple continuous states or pattern recognition can be set. For example, "the temperature exceeds the warning value three times in a row" will trigger an alarm to avoid false alarms caused by short-term fluctuations.

[0083] S63: Machine learning and deep learning applications: For historical time series data stored in Doris, feature engineering is performed according to time windows (such as days, weeks, and months), and data such as temperature, pressure, and vibration are aggregated into statistical features (maximum value, minimum value, average value, variance, etc.) to build the feature table required for model training; Python frameworks (such as TensorFlow and PyTorch) are used to train time series prediction models (such as LSTM or Transformer-based models) to predict the health status or failure possibility of equipment in the short term. During training, data from the past N days and N moments can be used as input to output estimated values ​​for a period of time in the future; after the model training is completed and the effect is verified, it is deployed as an online inference service, regularly obtaining the latest time series data from Doris, generating prediction results and comparing them with the current threshold to provide early warning of potential failures.

[0084] The beneficial effects of the present invention are:

[0085] 1. Significantly reduce data processing delay and improve real-time performance: Data is uploaded to Kafka in real time through the gateway and then consumed in parallel by Doris, reducing end-to-end processing delay from minutes to less than 30 seconds, and no more than 1 minute during peak hours; in high-concurrency scenarios, the data delivery success rate can reach 99.9%, effectively avoiding the "data window period" and achieving near-real-time monitoring and early warning; it solves the problem that traditional industrial systems rely heavily on batch imports, resulting in 5-10 minutes from on-site collection to queryability, and the lack of buffering can easily lead to data packet loss or backlog.

[0086] 2. Improve data quality and consistency, reducing missing and anomalies: Deduplication, interpolation, and filtering are completed before uploading, reducing the duplicate upload rate to less than 0.1%. The interpolation success rate for missing points exceeds 95%, and only blank values ​​are used to fill in missing periods exceeding 5 seconds. In the backend strict mode, the dirty data write rate is less than 0.5%, and the overall data integrity and consistency are significantly better than traditional solutions. This solves the problem of the direct database write mode in existing technologies that lacks deduplication, interpolation, and anomaly interception mechanisms, resulting in a common duplicate record rate of up to 2%, severe missing data and noise interference, and the need for subsequent manual cleanup and replenishment.

[0087] 3. Automated partitioning and storage cleanup reduces operation and maintenance costs and improves scalability: Utilizing Doris's dynamic partitioning function, the partitions from 3 days before the current day to 1 day after the current day are automatically maintained, and are set to be automatically deleted after 30 days without manual intervention; in actual operation, the data volume of a single partition is stably controlled at 50-80GB, and the query and write performance remains stable; when the number of devices increases, only the Kafka Broker and Doris BE nodes need to be expanded, and the operation and maintenance partition management workload is reduced by about 90%; it solves the problem that traditional database partitions need to be manually added, archived, and deleted. As the data accumulates, the workload is large. If partition management is omitted, the size of a single partition can exceed 100GB, leading to performance bottlenecks. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is a flowchart of the Doris method for storing device data based on the Industrial Internet proposed by the present invention. DETAILED DESCRIPTION

[0089] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0090] Please see the attached Figure 1 , a Doris method for storing device data based on the industrial Internet, including the following specific steps:

[0091] S1: Acquire industrial equipment node data: collect data from on-site PLC or gateway devices in real time to prepare for subsequent cleaning and storage;

[0092] S11: Collection method selection: PLC direct connection collection or gateway forwarding collection;

[0093] S111: PLC direct connection acquisition: The on-site PLC establishes a communication protocol such as Modbus (TCP / RTU), PROFINET, or EtherNet / IP with the host computer via Ethernet or serial port. The system initiates a read request to the PLC at a preset sampling frequency (for example, once per second or once every ten seconds) to obtain the real-time value of a specified register or data block. It is important to confirm the register range (start and end addresses) read during each acquisition. This is configured by the equipment manufacturer or project implementer based on actual needs.

[0094] S112: Gateway forwarding and collection: If some field devices do not support traditional PLC protocols, an industrial edge gateway (such as an OPC UA gateway, Modbus gateway, Siemens S7 gateway, etc.) is required to collect and parse the data of the device's own protocol. The parsed raw information (usually key-value pairs or a custom binary format) is then converted into a unified data format (such as JSON) and forwarded to the back-end system.

[0095] S12: Transmission protocol and upload method: MQTT upload or HTTP interface upload;

[0096] S121: MQTT upload: Using the lightweight MQTT protocol, the gateway or acquisition end publishes the processed data in the form of messages to the specified topic. Topics can be hierarchically organized according to "company number / factory number / workshop number / equipment number", for example: / CompanyA / Factory1 / Line2 / PLC_001; QoS (Quality of Service) is usually available at two levels: "at least once" or "exactly once". "At least once" ensures that messages will not be lost, but duplication may occur; "exactly once" consumes more resources; the message retention flag (retain) is generally set to "do not retain" to prevent existing subscribers from receiving outdated historical messages; information carried in the report includes device number, timestamp, and sensor readings; if the connection is disconnected or the server returns an error, the number of retries and retry interval can be set, and after multiple failed attempts, an alarm will be issued or the data will be written to a local backup to prevent data loss;

[0097] S122: HTTP interface upload: Send the collected data to the backend service through HTTP POST request; interface address example: http: / / <backend domain name>:<port> / api / v1 /

[0098] device / Data; specify Content-Type: application / in the HTTP request header

[0099] json, the request body is in standard JSON format. If a request fails due to network problems or backend exceptions, the client will try again according to the pre-set retry strategy (for example, up to three retries, each with a half-second interval). If all attempts fail, the error information is recorded in the local log and an alarm is triggered to notify the operation and maintenance personnel.

[0100] S13: Timestamp and unique identifier processing: Both the data collection end and the gateway must use a consistent time base, typically millisecond-level Unix timestamps, which are the number of milliseconds from January 1, 1970, to the current time. This ensures that data uploaded by different devices has a consistent time ordering. For device unique identification, if the device itself does not have a globally unique ID, a fixed prefix (such as the gateway number) can be concatenated with the device's local number on the gateway. For example, "GW01_PLC05" can serve as the device's globally unique ID. This ensures that devices are not confused during subsequent processing and storage.

[0101] S2: Preprocess the acquired data: remove duplicates, complete, remove noise, and standardize the original collected data to ensure the quality and consistency of the data subsequently stored;

[0102] S21: Duplicate data filtering and discarding invalid data;

[0103] S211: Duplicate data filtering: If the same device uploads the same timestamp and data value to the backend multiple times within a short period of time, it is considered duplicate data. After receiving the data, the system first checks whether the device number and timestamp are exactly the same as the most recent one. If they are consistent, the later data is directly discarded to avoid redundant storage.

[0104] S212: Invalid Data Discard: Messages uploaded by devices often contain a "status code" field to indicate whether the sampling status is normal. If the status code is not "normal," the data is invalid and should be discarded without further processing. Furthermore, for key physical quantities such as temperature and pressure, if the values ​​significantly exceed the reasonable range defined in the product manual or by the business (for example, if the temperature exceeds the rated upper limit or falls below the rated lower limit), these values ​​are also considered abnormal data and are discarded and recorded in the log.

[0105] S22: Data Missing Filling and Interpolation: To ensure the continuity of time series data, it is sometimes necessary to fill in the missing moments during the collection process. A common practice is to check the time span between two adjacent valid samples of the same device. If the time interval does not exceed the preset maximum interval (for example, within 5 seconds), the linear relationship between the two previous and subsequent valid values ​​is used to estimate and fill in the missing points. If the interval is too large, no interpolation is performed and the previous valid value or null value is directly filled in to avoid error accumulation. For simple cases, "nearest value filling" is used to directly replace the valid sample value closest to the missing moment. If higher accuracy is required, advanced interpolation algorithms (such as polynomial interpolation and spline interpolation) can also be used at the application layer. However, the implementation cost and computational overhead are relatively high, and they are rarely used in general industrial IoT scenarios.

[0106] S23: Data cleaning and standardization: including unit unification, noise filtering, and field renaming and supplementation;

[0107] S231: Unit Unification: All physical quantities must be converted to standard units. For example, if the temperature unit uploaded by a field device is in Fahrenheit, the system should convert it to Celsius before proceeding to the next step to ensure consistency in subsequent analysis and storage. The same approach applies to various physical quantities such as pressure, humidity, current, and voltage.

[0108] S232: Noise filtering: Due to noise interference during field acquisition, data smoothing is sometimes required. Common methods include sliding average or median filtering. For example, averaging the values ​​of the most recent N samples can eliminate short-term pulse interference, or taking the median of N samples to further remove the influence of extreme values.

[0109] S233: Field renaming and supplementation: If the downstream system (such as Kafka producers) requires fixed field names and formats, the original field names must be uniformly renamed. For example, the "temp" field can be changed to "temperature" after cleaning; "press" can be changed to "pressure". If the message lacks certain predefined fields, default values ​​can be added (such as a numeric value set to 0 or a string value set to "unknown") or set to null, depending on business needs.

[0110] S24: Data encapsulation into JSON: After completing the above cleaning, the data is organized into a unified JSON structure, which usually contains the following parts: device_id: unique device identifier;

[0111] timestamp: millisecond-level Unix timestamp when the server receives the data; data: key-value pairs of core sensor readings (such as temperature, pressure, humidity, etc.); status: device status code, used to indicate whether the sampling is normal; metadata: optional information, such as gateway number, signal strength, collection end IP, and other auxiliary fields;

[0112] Example JSON (illustrative only):

[0113] { "device_id": "GW01_PLC05", "timestamp": 171XXXXXXX000,

[0114] "data": { "temperature": 42.8, "pressure": 1.05, "humidity": 58.2}, "status": 0, "metadata": { "gateway_id": "GW01", "signal_strength": -65}};

[0115] After adopting this fixed format, when sending to Kafka or calling the HTTP interface, the JSON structured data can be directly used as the message body, keeping the meaning of each field consistent;

[0116] S3: Uploads pre-processed data to Kafka. This uses distributed message queues to achieve data buffering and decoupling, enabling parallel downstream consumption and good scalability.

[0117] S31: Kafka cluster and topic planning: Build at least three Kafka Brokers on the backend to ensure high system availability; design topic names according to business logic, such as "companyA_

[0118] "factory1_line2_equipment_data" aggregates data from the same production line or the same type of equipment under the same topic for centralized processing. Based on the peak message volume and system concurrency, configure an appropriate number of partitions for this topic. More partitions increase downstream parallel consumption capabilities, but also increase broker management overhead. Typically, the system's peak message rate (e.g., 100,000 messages per second) is evaluated, and then the maximum throughput of each partition is considered to determine an appropriate number of partitions, such as 8, 16, or 32. Set the replication factor to 3, meaning each message will have a replica on three brokers to ensure data readability even if any two nodes fail.

[0119] S32: Kafka Producer configuration;

[0120] S321: Broker list: write the addresses of all Brokers into the Producer configuration, for example broker1:9092,broker2:9092,broker3:9092;

[0121] S322: Message Acknowledgement Mechanism (acks): It is recommended to set it to "all", which means that the message is sent successfully only after it has been written to all In-Sync replicas, ensuring that no data is lost.

[0122] S323: Retry times: Automatically retry when sending fails, for example, set to 3 times; if it still fails after multiple retries, an alarm will be triggered or written to the local log to prevent silent data loss;

[0123] S324: Serializer: Use string serialization when sending the device number as a key, and also use UTF-8 encoding serialization when sending a JSON string as a value;

[0124] S325: Partition allocation strategy: The default hash partition strategy is used. This strategy uses the device number as the hash and then modulo the total number of partitions. This ensures that all data on the same device falls into the same partition, ensuring the sequentiality and sequential consumption of data on the same device.

[0125] S33: Message sending and exception handling: The Producer calls the asynchronous sending interface and monitors the sending result in the callback function: if the sending is successful, it will record the success log or ignore it; if the sending fails, the reason for the failure (such as network timeout, Broker unavailability, etc.) will be recorded in the log, and another attempt will be made according to the retry strategy; if it still fails after multiple retries, the message will be placed in the local temporary buffer queue and delivery will be attempted again after background recovery or manual intervention;

[0126] Throughput optimization: You can set appropriate batch sending parameters. For example, when a certain number of messages (such as 500) or a certain size of bytes (such as 32KB) are accumulated, they are sent all at once to avoid the network overhead caused by frequent small batch sending. You can also set "linger.ms" to tens of milliseconds to allow the Producer to bundle messages arriving within a short period of time before sending them.

[0127] S4: Doris consumes and stores Kafka data: Doris pulls JSON data from Kafka in real time and writes it to a relational table. It also uses dynamic partitioning to automate data partitioning and expiration cleanup.

[0128] S41: Doris table structure and dynamic partition design: Create a wide table in Doris, the fields include: device_id, event_time, temperature, pressure, humidity, status, gateway_id, etc.; the table is partitioned by date range (Range Partitioning) and enable dynamic partitioning; the system automatically creates five partitions based on the current date, from "3 days before the current day" to "1 day after the current day", with partition naming format pYYYYMMDD. For example, p20250606 represents the partition for June 6, 2025. This ensures that data for that day and recent historical days has corresponding partitions, making queries more efficient; set a partition retention period, such as 30 days. That is, when a partition is created for a certain number of days (such as 30 days), the system will automatically delete it, realizing automatic cleanup of expired data, which makes storage space management more flexible; at the same time, the table is bucketed (Hash bucketing), for example, hash bucketing is performed by device_id, with 16 buckets set. This allows data from different devices or partitions to be distributed to multiple nodes in parallel during queries and writes, improving performance;

[0129] S42: Create a Routine Load task: Execute a special SQL statement on Doris to create a Routine Load task, specifying which Kafka topic the task pulls data from, the concurrency rate to use, the maximum number of records or bytes to pull per batch, and how to parse JSON. The mapping of JSON fields to table fields is provided by a JSONPaths file. The corresponding relationship is roughly as follows: $.device_id corresponds to the device_id in the table; $.timestamp corresponds to event_time, and the timestamp needs to be converted to the Doris DATETIME type when writing; $.data.temperature, $.data.pressure, and $.data.humidity correspond to the temperature, pressure, and humidity fields in the table, respectively; $.status corresponds to the status in the table;

[0130] $.metadata.gateway_id corresponds to the gateway_id in the table. In the Routine Load task configuration, you can specify the number of concurrent consumers (for example, 4 concurrent threads), the maximum number of rows per batch (for example, 5000) or the maximum byte size (for example, 10MB), and the strict mode switch. When strict mode is enabled, if a field in a JSON entry is missing or the type does not match, it will be considered "dirty data" and discarded. Logs will also be recorded to ensure data consistency.

[0131] S43: Data loading process and dirty data processing: Doris Routine Load pulls messages from Kafka partitions in parallel according to the specified concurrency. Each message that meets the threshold (number of rows or bytes) is stored in a temporary file. The system decompresses and parses the temporary file, extracts fields from the JSON according to the JSONPaths mapping, and writes them to the memory table. The memory table data is automatically allocated to the corresponding partition according to the partitioning strategy, and then the backend node persists the data in batches to the column store. For records that fail to parse, have mismatched field types, or are missing required fields, Routine Load will place the record in the "dirty data" queue and decide whether to ignore or issue an alarm based on the "maximum dirty data ratio" threshold. If the dirty data ratio of a batch exceeds the threshold (for example, 5%), a system alarm will be triggered, and manual inspection will be conducted to check whether the JSON format or JSONPaths mapping is incorrect.

[0132] S44: Dynamic partitioning and expiration cleanup: Since the dynamic partitioning function is turned on, Doris will automatically determine the current date every day and create 5 partitions from "3 days before the current day" to "1 day after the current day". For example, if today is June 6, 2025, p20250603, p20250604, p20250605, p20250606, and p20250607 will be created; when a partition is created for a certain number of days (such as 30 days) (for example, after p20250506 expires), the system will automatically delete the partition without human intervention, thereby saving storage space; if the production environment needs to retain historical data for a longer period of time, just change the "partition expiration time" parameter from 30 days to 60 days, 90 days or longer;

[0133] S5: Monitoring and performance optimization: Monitor the end-to-end link from data collection to storage, promptly identify bottlenecks and anomalies, and perform targeted optimization.

[0134] S51: End-to-end monitoring indicators: including collection end monitoring, Kafka monitoring and Doris monitoring;

[0135] S511: Acquisition end monitoring:

[0136] Is the sampling frequency stable? Regularly count the number of samples per second or per minute. If the sampling frequency drops sharply or is interrupted, check the PLC connectivity or gateway status.

[0137] Network packet loss rate and retry times: When the gateway delivers to the backend, if the network condition is poor, retries or packet loss may occur. Monitor the ratio of retries to total transmission times and issue an alarm when it exceeds a certain threshold (such as 5%).

[0138] S512: Kafka Monitoring:

[0139] Consumer Lag: This measures the difference between the consumption speed of downstream consumers (Routine Load) and the sending speed of producers (Producers). Specifically, it checks the difference between the "Current Write Offset" and the "Committed Offset" of each partition. When the lag value increases and continuously exceeds a certain threshold (such as 10,000 records), it indicates that downstream consumption is insufficient and needs to be expanded or the number of concurrent users adjusted.

[0140] Throughput: Count the message writing rate (messages / second) and byte rate of the Producer to Kafka to confirm whether the maximum carrying capacity of the cluster has been reached;

[0141] Partition distribution: Check whether messages are evenly distributed across partitions. If the load on a partition is much higher than that on other partitions, consider increasing the number of partitions or using a custom partitioning strategy.

[0142] S513: Doris monitoring:

[0143] Routine Load Delay: This measures the time difference between the latest data in Kafka being written to Doris. You can regularly query the maximum event time (event_time) of the inserted data in the table and compare it with the current system time. If the delay lasts for more than 5 minutes, there may be a downstream write performance bottleneck, which requires troubleshooting.

[0144] Write QPS / TPS: Counts the number of queries per second or transactions per second for each BE node during write operations. If a node is continuously under high load (e.g., CPU usage exceeds 80%), consider capacity expansion or parameter optimization.

[0145] Partition size: Monitor the data volume of each partition. If the data volume of a partition far exceeds expectations (for example, exceeding 100GB), consider adjusting the number of buckets or further refining the historical partitions.

[0146] S52: Performance optimization strategy: including Kafka optimization and Doris optimization;

[0147] S521: Kafka optimization:

[0148] Batch sending and compression: Set an appropriate batch waiting time (such as tens of milliseconds) on the Producer side, aggregate multiple messages arriving within a short period of time and send them together. Enable compression algorithms such as Snappy or LZ4 to reduce network bandwidth usage and increase the Broker's write speed.

[0149] Properly configure the number of concurrent consumers: If the number of concurrent consumers in Routine Load is too low, the consumption speed will not keep up with the production speed; if the number of concurrent consumers is too high, Doris's write tasks will compete for resources on the backend nodes. It is necessary to find the optimal balance of concurrent consumers based on the CPU, memory, and disk I / O indicators of the BE node;

[0150] S522: Doris optimization:

[0151] Column encoding and compression: For commonly used fields in the table (such as temperature, pressure, and other numerical columns), column compression encoding methods (such as RLE and dictionary compression) are used to significantly reduce storage usage and improve scanning efficiency.

[0152] Small file merging: If frequent writing results in a large number of small files, which will affect query speed, you can perform partition file merging operations regularly to automatically merge small files into large files.

[0153] Statistics maintenance: Regularly perform analysis operations on tables and update column histogram statistics to enable the query optimizer to select a better execution plan.

[0154] S6: Extension: Secondary utilization and downstream analysis, while ensuring data quality and real-time performance, support subsequent data warehouse docking, real-time computing and machine learning applications;

[0155] S61: Data warehouse connection: In Doris, wide tables are constructed into fact tables, and then dimension tables are designed according to business needs. If more complex online analytical processing (OLAP) is required, data can be synchronized to downstream data warehouses (such as Hive / Hadoop, ClickHouse, Presto, etc.) through ETL or scheduled tasks for offline reporting and multidimensional analysis. In the same Doris cluster, views or hierarchical table structures can also be combined to provide data of different granularities to BI tools in a timely manner.

[0156] S62: Real-time computing and alerting: Using Apache Flink and Spark Streaming's streaming processing engines between Kafka and Dori, or in parallel with Doris, pre-processed data is directly consumed for complex event processing (such as anomaly detection and trend analysis). When indicators such as temperature and pressure exceed preset thresholds, SMS or email alerts are immediately triggered. Based on the CEP (complex event processing) model, multiple continuous states or pattern recognition can be set. For example, "the temperature exceeds the warning value three times in a row" will trigger an alarm to avoid false alarms caused by short-term fluctuations.

[0157] S63: Machine learning and deep learning applications: For historical time series data stored in Doris, feature engineering is performed according to time windows (such as days, weeks, and months), and data such as temperature, pressure, and vibration are aggregated into statistical features (maximum value, minimum value, average value, variance, etc.) to build the feature table required for model training; Python frameworks (such as TensorFlow and PyTorch) are used to train time series prediction models (such as LSTM or Transformer-based models) to predict the health status or failure possibility of equipment in the short term. During training, data from the past N days and N moments can be used as input to output estimated values ​​for a period of time in the future; after the model training is completed and the effect is verified, it is deployed as an online inference service, regularly obtaining the latest time series data from Doris, generating prediction results and comparing them with the current threshold to provide early warning of potential failures.

[0158] From the above description, it can be seen that the above-mentioned embodiments of the present invention achieve the following technical effects: significantly reduce data processing delay and improve real-time performance: upload to Kafka in real time through the gateway, and then consume in parallel by Doris, so that the end-to-end processing delay is reduced from minutes to less than 30 seconds, and no more than 1 minute during peak hours; in high-concurrency scenarios, the data delivery success rate can reach 99.9%, effectively avoiding the "data window period" and realizing near-real-time monitoring and early warning; solve the problem that traditional industrial systems rely more on batch imports, resulting in 5-10 minutes from on-site collection to query, and data packet loss or backlog is prone to occur when there is a lack of buffering.

[0159] Improve data quality and consistency, and reduce missing and anomalies: Deduplication, interpolation, and filtering are completed before uploading, reducing the duplicate upload rate to less than 0.1%, and the interpolation success rate of missing points exceeds 95%. Only empty values ​​are used to fill in missing periods exceeding 5 seconds. In the strict back-end mode, the dirty data write rate is less than 0.5%, and the overall data integrity and consistency are significantly better than traditional solutions. This solves the problem of the direct database write mode in existing technologies lacking deduplication, interpolation, and anomaly interception mechanisms, with a common duplicate record rate of up to 2%, severe lost data and noise interference, and the need for subsequent manual cleanup and replenishment.

[0160] Automated partitioning and storage cleanup reduce operation and maintenance costs and improve scalability: Utilizing Doris's dynamic partitioning function, the partitions from 3 days before the current day to 1 day after the current day are automatically maintained, and are set to be automatically deleted after 30 days without manual intervention; in actual operation, the data volume of a single partition is stably controlled at 50-80GB, and the query and write performance remains stable; when the number of devices increases, only the Kafka Broker and Doris BE nodes need to be expanded, and the operation and maintenance partition management workload is reduced by about 90%; it solves the problem that traditional database partitions need to be manually added, archived, and deleted. As the data accumulates, the workload is large. If partition management is omitted, the size of a single partition can exceed 100GB, leading to performance bottlenecks.

[0161] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Within the scope of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0162] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the specification. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Doris method for storing device data based on the Industrial Internet, characterized in that: The specific steps include: S1: Acquire industrial equipment node data: collect data from on-site PLC or gateway devices in real time to prepare for subsequent cleaning and storage; S2: Preprocess the acquired data: remove duplicates, complete, remove noise, and standardize the original collected data to ensure the quality and consistency of the data subsequently stored; S3: Uploads pre-processed data to Kafka. This uses distributed message queues to achieve data buffering and decoupling, enabling parallel downstream consumption and good scalability. S4: Doris consumes and stores Kafka data: Doris pulls JSON data from Kafka in real time and writes it to a relational table. It also uses dynamic partitioning to automate data partitioning and expiration cleanup. S41: Doris table structure and dynamic partition design: Create a wide table in Doris that is partitioned by date range and has dynamic partitioning enabled. The system automatically creates 5 partitions, from "3 days before the current date" to "1 day after the current date," based on the current date. Set the partition retention period. When a partition is created for a certain number of days, the system automatically deletes it. At the same time, perform bucketing on the table. S42: Create a Routine Load task: Execute a special SQL statement on Doris to create a Routine Load task, specifying which Kafka topic the task pulls data from, the concurrency number to use, the maximum number of records or bytes to pull per batch, and how to parse JSON; The mapping of JSON fields to table fields is provided by a JSONPaths file. In the RoutineLoad task configuration, you can specify the number of concurrent consumers, the maximum number of rows or bytes per batch, and the strict mode switch. S43: Data loading process and dirty data processing: Doris Routine Load pulls messages from Kafka partitions in parallel according to the specified concurrency. Each message that meets the threshold is stored in a temporary file. The system decompresses and parses the temporary file, extracts fields from the JSON according to the JSONPaths mapping, and writes them to the memory table. The memory table data is automatically allocated to the corresponding partition according to the partitioning strategy, and then the backend node persists the data in batches to the column store. For records that fail to parse, have mismatched field types, or are missing required fields, Routine Load will place the record in the "dirty data" queue and decide whether to ignore or issue an alarm based on the "maximum dirty data ratio" threshold. If the dirty data ratio of a batch exceeds a certain value, a system alarm will be triggered, and manual inspection will be conducted to check whether the JSON format or JSONPaths mapping is incorrect. S44: Dynamic partitioning and expired cleanup: Since the dynamic partitioning function is turned on, Doris will automatically determine the current date every day and create 5 partitions from "3 days before the current day" to "1 day after the current day"; when a partition is created for a certain number of days, the system will automatically delete the partition; S5: Monitoring and performance optimization: Monitor the end-to-end link from data collection to storage, promptly identify bottlenecks and anomalies, and perform targeted optimization. S51: End-to-end monitoring indicators: including collection end monitoring, Kafka monitoring and Doris monitoring; S52: Performance optimization strategy: including Kafka optimization and Doris optimization; S6: Extension: Secondary utilization and downstream analysis, while ensuring data quality and real-time performance, support subsequent data warehouse docking, real-time computing and machine learning applications; S61: Data warehouse connection: In Doris, construct wide tables into fact tables, and then design dimension tables according to business needs. If more complex online analytical processing is required, data can be synchronized to downstream data warehouses through ETL or scheduled tasks for offline reporting and multi-dimensional analysis; In the same Doris cluster, you can also combine views or hierarchical table structures to provide data of different granularities to BI tools in a timely manner; S62: Real-time computing and alerting: Using stream processing engines like Apache Flink and Spark Streaming between Kafka and Dori, or in parallel with Doris, pre-processed data is directly consumed and then subjected to complex event processing. When indicators such as temperature and pressure exceed preset thresholds, SMS or email alerts are immediately triggered. Based on the CEP model, multiple continuous states or pattern recognition can be set. S63: Machine learning and deep learning applications: For the historical time series data stored in Doris, feature engineering is performed according to the time window, the data is aggregated into statistical features, and the feature table required for model training is constructed; the Python framework is used to train the time series prediction model to predict the health status or failure possibility of the equipment in the short term. During training, the data of the past N days and N moments can be used as input to output the estimated value for a period of time in the future; after the model training is completed and the effect is verified, it is deployed as an online inference service, regularly obtaining the latest time series data from Doris, generating prediction results and comparing them with the current threshold to provide early warning of potential failures.

2. The Doris method for storing device data based on the industrial Internet according to claim 1 is characterized in that: Said S1 specifically includes: S11: Collection method selection: PLC direct connection collection or gateway forwarding collection; S12: Transmission protocol and upload method: MQTT upload or HTTP interface upload; S13: Timestamp and unique identification processing: A consistent time base should be used at the collection end and the gateway, generally using millisecond-level Unix timestamps; for the unique identification of the device, if the device itself does not have a globally unique ID, it can be composed of a fixed prefix and the device's local number on the gateway.

3. The Doris method for storing device data based on the Industrial Internet according to claim 2 is characterized in that: The S12 specifically includes: S121: MQTT upload: Using the lightweight MQTT protocol, the gateway or acquisition end publishes processed data in the form of messages to designated topics. Topics can be hierarchically organized according to "Enterprise ID / Factory ID / Workshop ID / Equipment ID." QoS typically offers two levels: "At least once" and "Exactly once." The message retention flag is typically set to "Do not retain." If the connection is lost or the server returns an error, the number of retries and retry interval can be set. After multiple failed attempts, an alarm will be generated or the message will be written to a local backup. S122: HTTP interface upload: The collected data is sent to the backend service through an HTTP POST request; the Content-Type is specified in the HTTP request header; if the request fails due to network problems or backend exceptions, the client will try again according to the pre-set retry strategy; if all attempts fail, the error information is recorded in the local log and an alarm is triggered to notify the operation and maintenance personnel.

4. The Doris method for storing device data based on the industrial Internet according to claim 1 is characterized in that: The S2 specifically includes: S21: Duplicate data filtering and discarding invalid data; S22: Data Missing Filling and Interpolation: Checks the time span between two consecutive valid samples of the same device. If the time interval does not exceed the preset maximum interval, the linear relationship between the two valid values ​​is used to estimate and fill the missing point. If the interval is too large, no interpolation is performed and the previous valid value or null value is directly filled to avoid error accumulation. For simple cases, "nearest value filling" is used to directly replace the valid sample value closest to the missing time. For higher accuracy, advanced interpolation algorithms can also be used at the application layer. S23: Data cleaning and standardization: including unit unification, noise filtering, and field renaming and supplementation; S24: Data encapsulation as JSON: After completing the above cleaning, organize the data into a unified JSON structure, which usually contains the following parts: device_id, timestamp, status, metadata; after adopting this fixed format, when sending to Kafka or calling the HTTP interface, the JSON string can be directly used as the message body to keep the meaning of each field consistent.

5. The Doris method for storing device data based on the industrial Internet according to claim 1 is characterized in that: The S3 specifically includes: S31: Kafka Cluster and Topic Planning: Build at least three Kafka brokers on the backend, design topic names based on business logic, and aggregate data from the same production line or the same type of equipment under the same topic. Configure an appropriate number of partitions for the topic based on peak message volume and system concurrency. Set the replication factor to 3, so that each message will have a replica on all three brokers. S32: Kafka Producer configuration; S33: Message sending and exception handling: Producer calls the asynchronous sending interface and monitors the sending result in the callback function: if the sending is successful, it will record the success log or ignore it; if the sending fails, the reason for the failure will be recorded in the log, and another attempt will be made according to the retry strategy; if it still fails after multiple retries, the message will be placed in the local temporary buffer queue and delivery will be attempted again after background recovery or manual intervention.

6. The Doris method for storing device data based on the industrial Internet according to claim 5 is characterized in that: The S32 specifically includes: S321: Broker list: write the addresses of all Brokers into the Producer configuration; S322: Message confirmation mechanism: It is recommended to set it to "all", which means that the message is sent successfully only after it is written to all In-Sync replicas to ensure that data is not lost; S323: Retry times: Automatically retry when sending fails. If it still fails after multiple retries, an alarm will be triggered or written to the local log; S324: Serializer: Use string serialization when sending the device number as a key, and also use UTF-8 encoding serialization when sending a JSON string as a value; S325: Partition allocation strategy: Use the default hash partition strategy, hash the device number and then modulo the total number of partitions.

7. The Doris method for storing device data based on the Industrial Internet according to claim 6 is characterized in that: The S52 specifically includes: S521: Kafka optimization: Batch sending and compression: Set an appropriate batch waiting time on the Producer side, aggregate multiple messages arriving within a short period of time and send them together, and enable the compression algorithm; Reasonably configure the number of concurrent consumers: Find the optimal balance of concurrent consumers based on the CPU, memory, and disk I / O indicators of the BE node; S522: Doris optimization: Column encoding and compression: Column compression encoding is used for commonly used fields in the table, significantly reducing storage usage and improving scanning efficiency. Small file merging: Partition file merging operations can be performed regularly, allowing small files to be automatically merged into large files; Statistics maintenance: Regularly perform analysis operations on tables to update column histogram statistics, allowing the query optimizer to select a better execution plan.

Images