Mass process data presentation method based on Websocket

Abstract

The invention relates to the technical field of computer data processing, and discloses a Websocket-based massive process data presentation method, which comprises the following steps of: writing process data points into a Dores distributed database cluster; based on the historical data query request, generating a cache key and querying the Redis memory database, if the query result is not hit, querying the Dores distributed database cluster to obtain a query result set, and writing the result set into the Redis memory database; and receiving a subscription message through the Websocket communication connection, and when a new process data point is received, pushing the new process data point to a front-end visual display layer. According to the method and the system, the data architecture combining the distributed database and the memory database is constructed, and the Websocket communication mechanism is used for pushing the real-time data, so that the problems of slow historical data query response and high real-time data presentation delay are solved, and the query performance and the real-time performance of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, specifically to a method for presenting massive amounts of process data based on WebSocket. Background Technology

[0002] In modern industrial production processes, various sensors and automated equipment generate massive amounts of industrial process data. Effective storage, retrieval, and visualization of this data are fundamental for production process monitoring, fault diagnosis, and process optimization.

[0003] When dealing with massive amounts of industrial process data, existing data presentation technologies often require direct access to the underlying database when users need to query historical data over a long period. Due to the sheer volume of data, the query operation is time-consuming, causing the front-end interface to wait for extended periods, which impacts the efficiency of data analysis.

[0004] In real-time data monitoring, existing technologies use front-end polling to obtain the latest data. The front-end polling mechanism requires the client to periodically send HTTP requests to the server. This method not only increases the server load and network overhead, but also makes it difficult to balance data update frequency and response time, resulting in data presentation delays and screen stuttering, which cannot meet the needs of high real-time monitoring scenarios.

[0005] Furthermore, with the development of the Industrial Internet, the number of data sources and the total amount of data continue to grow. Some existing system architectures lack horizontal scalability and are prone to performance bottlenecks when faced with large-scale device access. At the same time, the ability to integrate and compare data from different process units within a single view is also insufficient, limiting the breadth and depth of data analysis.

[0006] Therefore, this invention proposes a method for presenting massive amounts of process data based on WebSocket to address the shortcomings of existing technologies. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a WebSocket-based method for presenting massive amounts of process data. This method solves the problems of long response times for historical data queries and high latency for real-time data push in high-concurrency scenarios when processing massive amounts of industrial process data, making it difficult to simultaneously meet the needs of large-scale data storage and analysis with low-latency real-time presentation.

[0008] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a method for presenting massive amounts of process data based on WebSocket, comprising the following steps: Write process data points generated from industrial data sources into a Doris distributed database cluster; Based on a historical data query request initiated by a front-end visualization layer, a cache key is generated, and a Redis in-memory database is queried based on the cache key; When no data corresponding to the cache key is found in the Redis in-memory database, the Doris distributed database cluster is queried to obtain a query result set. The query result set and the cache key are written into the Redis in-memory database and the query result set is sent to the front-end visualization layer. Through a Websocket communication connection established with the front-end visualization layer, a subscription message containing one or more process measurement point identifiers is received; When a new process data point is received, based on the process measurement point identifier contained in the new process data point and the subscription message, the new process data point is pushed to the front-end visualization layer through the Websocket communication connection.

[0009] This invention achieves efficient data storage and query acceleration by constructing a data storage model specifically optimized for time-series data. Process data points are stored in a Doris data table using the DUPLICATE KEY data model, and a composite sorting key consisting of an equipment identifier field, a measurement point identifier field, and a timestamp field is configured for the Doris data table. This design leverages the principle of locality of reference in physical data storage, improving the efficiency of querying time ranges for specific equipment and measurement points. Furthermore, a time range partitioning strategy and a hash bucketing strategy based on the equipment identifier field are employed to distribute and store process data points across multiple BE nodes in the Doris distributed database cluster. This strategy enables the system to perform partitioning, pruning, and parallel computation during query execution, shortening the retrieval time for massive amounts of historical data.

[0010] This invention provides a high-throughput, low-latency data writing path for real-time data streams. Through a Stream Load mechanism, external data can be efficiently distributed to target BE nodes for parallel writing via FE nodes and coordinating nodes in the Doris distributed database cluster, in batches. This mechanism ensures the stable and rapid persistence of massive amounts of real-time data.

[0011] This invention establishes a "bypass caching" data access mode, reducing the response latency of high-frequency data requests. For historical data queries, the system first accesses the Redis in-memory database. Only in the event of a cache miss does the system backtrack to the Doris distributed database cluster. This mode handles the vast majority of the query load with the in-memory database, avoiding repeated access to the underlying storage. By setting a predetermined lifespan for the cache key, specifically by setting a predefined validity period and calculating the absolute expiration timestamp, this method establishes an automated cache invalidation mechanism, ensuring data timeliness while preventing the unlimited growth of cached data.

[0012] This invention designs a precise and efficient cache management strategy. By concatenating a fixed string prefix, device identifier, measurement point identifier, start timestamp, and end timestamp into a unique string as the cache key, it ensures that each independent historical data query request can be accurately mapped to a cache entry, avoiding data confusion. Simultaneously, optimized Redis data structures are employed for different data access patterns. For example, a Redis Hash data structure is used for a single latest real-time process data point, while a Redis Sorted Set data structure is used for short-cycle historical trend data sequences. This design improves cache query efficiency while also optimizing memory usage.

[0013] This invention implements a precise real-time data push mechanism. Through a WebSocket communication connection, the backend service maintains a subscription table recording client connections and subscription relationships. When a process data point with a timestamp later than the current time is received (i.e., a new process data point is received), the backend service only pushes the data to client connections that have explicitly subscribed to the process measurement point identifier corresponding to this data point. This avoids broadcasting invalid data, reduces network bandwidth consumption and client processing load, and ensures low latency for real-time data push under a large number of concurrent client connections.

[0014] This invention achieves dynamic and smooth data visualization on the front end. The front-end visualization layer distinguishes between real-time and historical data, implementing two independent rendering paths: incremental rendering upon receiving new process data points, and full rendering upon receiving a query result set. This separate rendering strategy ensures smooth updates of real-time curves and rapid plotting of historical curves, enhancing the user experience.

[0015] Furthermore, this invention provides an active update mechanism as a supplement to the caching strategy. A separate background task actively queries and writes or overwrites core hot data into the Redis in-memory database at preset intervals. This mechanism ensures the highest timeliness of critical business data and meets stringent real-time monitoring requirements.

[0016] This invention provides a method for presenting massive amounts of process data based on WebSocket. It has the following beneficial effects: 1. This invention achieves parallel scanning of distributed storage data by employing a time-range partitioning strategy and a hash bucketing strategy in a Doris distributed database cluster, reducing the reading of irrelevant data. Simultaneously, a data caching layer using a Redis in-memory database is set up, and a bypass caching mode is employed to temporarily store the result sets of historical data queries. This allows high-frequency data requests to be retrieved directly from memory, shortening the data query response time and improving the query response speed for massive amounts of historical process data.

[0017] 2. This invention utilizes the WebSocket communication mechanism to establish a persistent bidirectional communication connection. The backend business service module can proactively push new process data points to the frontend visualization layer, avoiding the delay of frontend polling. After receiving new process data points, the frontend visualization layer performs incremental rendering operations through the AntV chart framework, updating only the changed parts of the chart. This ensures high smoothness of the interface display under continuous data updates, achieving low-latency real-time push and high-smoothness visualization of process data.

[0018] 3. The front-end visualization layer of this invention can be configured with multiple independent data series. Based on the equipment identifier and measurement point identifier fields contained in the data points, data from different sources is routed to the corresponding data series for rendering, enabling synchronous display and comparative analysis of data from different process units within a single chart view. The Doris distributed database cluster and Redis in-memory database used by the system both support horizontal scaling, achieving integrated display of multi-source data and possessing excellent system scalability, capable of adapting to the continuously growing needs of the number of connected devices and the amount of data in industrial internet scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system block diagram of the present invention; Figure 3 This is a comparison chart of historical data query response times in a specific application embodiment of the present invention; Figure 4 This is a comparison chart of the average latency of real-time data updates in a specific application embodiment of the present invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See attached document Figure 2 This invention provides a method for presenting massive amounts of process data based on WebSockets. This method is implemented using a WebSocket-based massive process data presentation system. The WebSocket-based massive process data presentation system adopts a layered design, divided into a data aggregation and storage layer, a data caching layer, and a front-end visualization layer. These three layers are logically independent and interact through standardized data interfaces, collectively forming a complete data processing and presentation system.

[0022] The data aggregation and storage layer forms the underlying data foundation of the entire system, responsible for the persistent storage and analysis of massive amounts of process data. This layer is built using a Doris distributed database cluster to receive and store process data from one or more industrial data sources. The Doris distributed database cluster provides upper-layer applications with the ability to query and aggregate all the data.

[0023] A data caching layer is deployed between the data aggregation and storage layer and the front-end visualization layer to improve the response speed of high-frequency data access. The data caching layer is implemented using a Redis in-memory database, specifically for temporarily storing hot process data and result sets of recent historical trend queries. By setting up a data caching layer, the direct access pressure on the data aggregation and storage layer can be reduced, and the response time for data requests can be shortened.

[0024] The front-end visualization layer is the application layer directly facing the user, responsible for the final presentation and interactive operation of process data. It establishes a persistent, bidirectional communication connection with the system backend service via a WebSocket mechanism to receive real-time pushed process data. The front-end visualization layer also integrates an AntV charting framework to render the received real-time data and historical data returned from queries into multi-dimensional visualization charts, such as real-time curves, historical trend charts, and dashboards.

[0025] See attached document Figure 1 The present invention provides a massive process data presentation system based on Websocket. The data processing flow of the system is divided into real-time data stream processing path and historical data stream processing path according to the timeliness of the data.

[0026] The real-time data stream processing path begins with the data acquisition equipment in the industrial field. This equipment periodically generates process data points containing timestamps, equipment identifiers, measurement point identifiers, and measurement point values. These process data points are then sent to the data aggregation and storage layer via a data interface.

[0027] The data aggregation and storage layer receives process data points and writes them into a pre-defined Doris database table. Simultaneously, the data synchronization module updates the latest process data points to the Redis in-memory database of the data caching layer for subsequent fast access.

[0028] The WebSocket service module continuously monitors data updates. When a new process data point arrives, the WebSocket service module retrieves all client connections that have subscribed to this identifier from the subscription relationship mapping table, based on the device identifier and measurement point identifier of the process data point.

[0029] The WebSocket service module proactively pushes new process data points to all corresponding front-end visualization layers by retrieving all client connections. Upon receiving the new process data points, the front-end visualization layer performs incremental rendering, updating the displayed content without redrawing the entire chart.

[0030] The historical data stream processing path is triggered by user interaction in the front-end visualization layer. After the user selects one or more measurement point identifiers, a start time, and an end time on the interface, the front-end visualization layer initiates a historical data query request to the application's back-end service.

[0031] After receiving a historical data query request, the application backend service first generates a cache key based on the request parameters, and then uses the cache key to query the data cache layer. The cache key is generated as follows: ; in, The generated cache key is a unique string; This is a deterministic function used to generate cache keys, such as a string concatenation function; It is a fixed string prefix used for namespace isolation; The device identifier specified in the historical data query request; The identifier of the measurement point specified in the historical data query request; The start timestamp specified in the historical data query request; The end timestamp specified in the historical data query request.

[0032] If the data caching layer contains data corresponding to the cache key, the cached data is returned directly. If it does not exist, the application backend service converts the historical data query request into an SQL query statement and sends it to the data aggregation and storage layer.

[0033] The Doris cluster in the data aggregation and storage layer executes SQL queries, scans data in distributed storage in parallel, and returns the query result set to the application backend service. Before returning the query result set to the frontend visualization layer, the application backend service first writes the query result set and the generated cache key to the data cache layer, and sets a predetermined lifecycle for the cache key.

[0034] Finally, the application backend service sends the query result set to the frontend visualization layer. After receiving the complete result set, the frontend visualization layer performs a full rendering operation to draw a complete historical trend chart on the interface.

[0035] See attached document Figure 2 The implementation of the data aggregation and storage layer relies on a data model and storage strategy optimized for the characteristics of massive industrial process data.

[0036] In the data aggregation and storage layer, all process data is stored in one or more tables within a Doris distributed database cluster. Each Doris table contains the following core fields: a timestamp field to record the data acquisition time, an equipment identifier field to uniquely identify the data source device, a measurement point identifier field to distinguish specific process parameters, and a measurement point value field to store the specific numerical value.

[0037] Doris tables use the DUPLICATE KEY data model. This model allows for the physical storage of completely duplicated data rows. Doris tables are configured with a composite sort key, which is formed by sequentially combining a device identifier field, a measurement point identifier field, and a timestamp field. Based on this composite sort key setting, data from the same device and the same measurement point are organized contiguously in physical storage. This data organization method improves the locality and efficiency of data retrieval when performing queries targeting specific devices and time ranges.

[0038] To manage massive amounts of time-series data and optimize query performance, Doris tables employ a time-range partitioning strategy. This strategy divides the table into multiple logical partitions based on the values ​​of the timestamp field, such as creating independent data partitions by "day" or "month". When a query request includes a time range condition, Doris' query planner performs partition pruning, scanning only the data partitions that intersect with the query's time range, thereby reducing the amount of irrelevant data scanned.

[0039] To achieve horizontal scaling of data and parallel query processing, Doris tables further employ a hash bucketing strategy. This strategy distributes data rows within each data partition into a predetermined number of buckets based on the value of the device identifier field. The bucket to which a data row belongs is determined by the following formula: ; in, A unique identifier for the target bucket to which the data row belongs; This is a hash function used to calculate the hash value of the input value; The value of the bucket key used for hash calculation is, in this embodiment, the actual value of the device identifier field within the data row; The total number of buckets pre-defined for Doris data tables.

[0040] Through a hash bucketing strategy, data shards are evenly distributed across the various Backend (BE) nodes in the Doris cluster. When performing complex computational tasks such as aggregation queries, query requests are distributed to all BE nodes holding the relevant data shards for parallel execution, thereby achieving high-performance analysis of massive amounts of data.

[0041] See attached document Figure 2 The data writing and aggregation mechanism provides two independent implementation paths for massive process data to enter the data aggregation and storage layer, which are used to process real-time data streams and batch historical data, respectively.

[0042] For writing real-time data streams, this invention employs a Stream Load mechanism. The external data acquisition system encapsulates the acquired process data into a data batch and sends a data load request to a FE (Frontend) node in the Doris cluster via the HTTP protocol. The data body of this request is in JSON or CSV format and contains multiple process data records to be written.

[0043] Upon receiving a data loading request, the FE node first performs authorization verification and request parsing. Then, it selects a BE (Backend) node from the BE (Backend) node list as the coordinating node for this loading task. The FE node returns the coordinating node's network address information to the external data acquisition system. The external data acquisition system then establishes a connection with the designated coordinating node and sends the complete data batch directly to this coordinating node.

[0044] After receiving a data batch, the coordinating node calculates the target BE node to be written to for each data record within the batch based on the bucket key (device identifier field) of the target Doris data table. The coordinating node then distributes the data records to the corresponding target BE nodes. Each target BE node, upon receiving the data record under its jurisdiction, performs a local write operation to persist the data to the storage engine.

[0045] The commit operation of a Stream Load task is transactional. The commit of a data batch is triggered by either the data size or the waiting time. This triggering condition can be defined by the following logical expression: ; in, This indicates the commit decision for the data batch; when the expression evaluates to true, the commit operation is executed. This represents the number of data rows or bytes accumulated in the current data batch. The preset threshold for the amount of data that triggers submission; This indicates the time elapsed since the last submission or the start of the task; This is the preset time threshold for triggering submission.

[0046] For importing batch historical data, this invention employs a Broker Load mechanism. This mechanism is used to load data files from external storage systems (such as HDFS or S3 object storage). The user submits a LOAD command to the FE node via a MySQL client, containing the data source path, the target Doris table name, and the data format definition.

[0047] After receiving and parsing the LOAD command, the FE node creates an asynchronous load job and generates a distributed execution plan. The FE node then distributes the data fetch and write tasks to multiple BE nodes in the cluster. The BE nodes, through their Broker processes, fetch data files from the external storage system in parallel, parse and transform the data, and write the transformed data to local storage. The FE node monitors the execution status of the entire Broker Load job until all data has been loaded.

[0048] See attached document Figure 2 The data caching layer adopts a "Cache-Aside" working mode, which defines a clear data query and backfill process for handling data requests from the front-end visualization layer.

[0049] When a backend business service module receives a data query request, it first generates a unique cache key based on the parameters in the data query request (such as the device identifier field, the measurement point identifier field, and the time range).

[0050] This backend business service module uses the generated unique cache key to initiate a read operation to the Redis in-memory database. If data corresponding to the unique cache key exists in the Redis in-memory database, this event is defined as a "cache hit". In a cache hit event, the backend business service module directly returns the data read from the Redis in-memory database as the response result to the initiator of the data query request, and the entire query process terminates.

[0051] If the Redis in-memory database does not contain data corresponding to the unique cache key, this event is defined as a "cache miss". In a cache miss event, the backend business service module forwards the original data query request to the Doris distributed database cluster in the data aggregation and storage layer.

[0052] The Doris distributed database cluster executes the query operation and returns the result dataset to the backend business service module. Upon receiving the result dataset, the backend business service module performs two steps: first, it associates the result dataset with the previously generated unique cache key and writes this key-value pair to the Redis in-memory database; second, it returns the result dataset as a response to the party that initiated the data query request.

[0053] The data source selection logic for a data query request can be determined by the following function: ; in, This indicates the data source selected to handle a data query request; This represents the function for selecting the data source. This represents a unique cache key generated based on the parameters of the data query request; This indicates that the data source is a Redis in-memory database; This indicates that the data source is a Doris distributed database cluster; This represents a Boolean function used to determine whether a cached key exists in the Redis in-memory database.

[0054] See attached document Figure 2 The data caching layer is designed for different types of process data, and specific Redis data structures are used to optimize storage space and query efficiency.

[0055] For a single, up-to-date real-time process data point, this invention employs a Redis hash data structure for caching. The value of a device identifier field is used as the key in the Redis hash. Internally, the value of a measurement point identifier field is used as a field, and the corresponding measurement point value field is stored as the value of this field. Using this structure, the latest real-time values ​​for all measurement points under a single device can be retrieved in a single operation.

[0056] For a short-term historical trend data sequence, this invention uses Redis's Sorted Set data structure for caching. A combined string containing a device identifier field and a measurement point identifier field is used as the key of the Sorted Set. Within the Sorted Set, the timestamp of each process data point is converted into a numerical form and used as the score for that data point. The combination of the data point's value and timestamp is serialized into a string, used as the member of that data point. This structure supports efficient retrieval of subsets of data points by timestamp range.

[0057] All cached data written to the Redis in-memory database uses a cache key generated using a unified rule. The generation of the cache key ensures that each individual query request maps to a unique cache entry. The rule for generating a cache key is defined by the following formula: ; in, This represents the final cache key string generated for locating data in Redis; This represents a predefined constant string prefix used to distinguish different cached data types; This indicates the value of the device identifier field specified in the data query request; This indicates the value of the measurement point identifier field specified in the data query request; Indicates the start timestamp of the time range specified in the data query request; Indicates the end timestamp of the time range specified in the data query request; This represents a string concatenation operator used to join multiple string parts into a single complete string.

[0058] For queries that do not involve a time range, such as requests to retrieve a single latest data point, the T_{start} and T_{end} parts are omitted when generating the cache key.

[0059] See attached document Figure 1 and attached Figure 2The data caching layer employs a synchronization and invalidation mechanism that combines passive invalidation with active updating to ensure the timeliness of cached data in the Redis in-memory database.

[0060] This invention employs a passive expiration strategy based on Time-To-Live (TTL). When a backend business service module performs a data write operation in "bypass caching" mode, it sets a predefined expiration time for the cache key while writing a cached key-value pair to the Redis in-memory database.

[0061] The absolute expiration timestamp of a cache key is calculated using the following formula: ; in, This represents the absolute expiration timestamp of a cached key in the Redis in-memory database; This represents the system timestamp indicating when the current data was written to the Redis in-memory database. This indicates the preset cache validity period for a specific type of data; it is a fixed time interval value.

[0062] Redis in-memory database reaches this absolute expiration timestamp When this happens, the corresponding cached key-value pair will be automatically deleted from memory. Afterward, any read request for the same cached key will result in a "cache miss," triggering a process to reload the latest data from the Doris distributed database cluster, thus updating the cache.

[0063] As a supplement, this invention also includes a proactive update mechanism. An independent background task, running at a preset cycle, retrieves a set of query request parameters defined as "hot data" from a configuration list. This background task uses these query request parameters to proactively initiate data queries to the Doris distributed database cluster. After obtaining the latest data results, the background task directly writes or overwrites these data results into the corresponding cache key in the Redis in-memory database and resets the cache key's expiration time. This proactive update mechanism ensures the timeliness of core data.

[0064] See attached document Figure 1 and attached Figure 2 The WebSocket real-time communication mechanism in the front-end visualization layer is used to establish a persistent bidirectional communication channel between the front-end and back-end business service modules to enable real-time push of process data.

[0065] When a user loads a visual view on the front-end page, the front-end client application initiates a WebSocket connection request to a predefined network endpoint of the back-end business service module. The back-end business service module receives this connection request and completes the WebSocket protocol handshake, thereby establishing a full-duplex communication connection between the client and the server.

[0066] Once the connection is successfully established, the front-end client application immediately sends a subscription message to the back-end business service module through this communication connection. This subscription message is a JSON-formatted text data frame, containing a list of one or more process measurement point identifiers. Each process measurement point identifier uniquely corresponds to a process data point that the front-end page needs to monitor in real time.

[0067] The backend business service module internally maintains a subscription relationship table. This table records the mapping relationship between each active WebSocket connection and the set of process measurement point identifiers subscribed to by that connection. When the backend business service module receives a subscription message from a client, it updates this subscription relationship table, associating the client's connection identifier with the list of process measurement point identifiers contained in the message.

[0068] When a new process data point (a process data point with a timestamp later than the current time) arrives at the backend business service module from the data source, the backend business service module first extracts the process measurement point identifier from this process data point. Then, the backend business service module queries the subscription relationship table to retrieve all client connections that have subscribed to this process measurement point identifier. Finally, the backend business service module encapsulates the new process data point into a JSON-formatted WebSocket data frame and sends it to all retrieved clients through the corresponding WebSocket connections.

[0069] The execution conditions for a data push decision are determined by the following logical function: ; in, This refers to a process data point. Connect to a client The push decision function yields the following result: This indicates that the push notification was executed, and the result is... This means that the push notification will not be executed; This indicates a newly arrived process data point; This indicates an active client WebSocket connection; Represents a data point used to process data. A function to extract unique process measurement point identifiers; Indicates connection with client A set of associated, subscribed process measurement point identifiers; This represents a set ownership check operator, used to determine whether an element exists within a set.

[0070] To maintain the stability of the WebSocket connection, the backend service module periodically sends heartbeat frames to each client connection. If no response frame is received from the client within a preset time period, the backend service module will proactively disconnect the connection. The frontend client application includes disconnection reconnection logic, which automatically attempts to re-establish the WebSocket connection when a connection interruption is detected.

[0071] See attached document Figure 1 and attached Figure 2 The front-end visualization layer utilizes the AntV chart framework to dynamically render real-time and historical data, and supports the fusion display of multi-source data.

[0072] When a front-end WebSocket client's `onmessage` event handler receives a real-time process data point pushed from the backend, the event handler parses the data point content and locates the corresponding chart instance based on the device identifier and measurement point identifier fields within the data point. This event handler then appends the new data point to the chart instance's data model. The AntV chart framework performs an incremental rendering operation, updating only the visualization elements in the chart related to the new data point.

[0073] When a user selects a time range and initiates a historical data query via an interface element, the frontend sends a data query request to the backend business service module via an HTTP API. The backend business service module returns a complete dataset containing all data points within the specified time range. After receiving the complete dataset, the frontend passes it to a chart instance, triggering a full rendering operation to draw the complete historical data curve.

[0074] To simultaneously display data from different devices or measurement points within a single chart view, a chart instance is configured to contain multiple independent data series. Each independent data series is associated with a unique data source identifier and assigned independent visual attributes, such as line color or data point shape. Whether the data points are real-time data points received via WebSocket or historical datasets obtained via HTTP API, the front end routes the data to the corresponding data series for rendering based on the device identifier and measurement point identifier fields contained within the data points.

[0075] The final visualization of a chart instance is the collection of rendering results for all its data series. This relationship can be defined by the following formula: ; This indicates the visualization state of the final generated chart; This represents the rendering function of the AntV chart framework, which transforms the dataset into a visual graph. Indicates the first A dataset bound to an independent data series, which contains one or more data points; This indicates the total number of independent data series configured in a single chart instance.

[0076] The AntV charting framework provides built-in interactive features, including chart zooming, chart panning, and data point tooltips. Users can use the chart zooming and panning functions to adjust the timeline range of the chart display. When a user hovers the cursor over a data point in the chart, the data point tooltip displays the precise timestamp and value of that data point.

[0077] Specific application examples: To more clearly illustrate the technical solution of this invention, a specific industrial application scenario will be used as an example for detailed explanation below.

[0078] Example 1: Multi-parameter monitoring and historical trend analysis of chemical reactors The application scenario of this embodiment is a chemical production workshop, which requires continuous real-time monitoring and historical data analysis of the internal pressure (measuring point marked P-101) and internal temperature (measuring point marked T-101) of a key piece of equipment—a reactor (equipment marked R-101). The data acquisition system generates and reports the process data points of P-101 and T-101 at a frequency of once per second.

[0079] The real-time monitoring process is as follows: Step 1: Data Generation and Import The pressure and temperature sensors on site generate one data record per second, containing a timestamp, device identifier: R-101, measurement point identifier: P-101 / T-101, and measurement point value. The data acquisition gateway encapsulates these data records into JSON format data batches and continuously writes them to the Doris database table in the data aggregation and storage layer through a Stream Load mechanism.

[0080] Step 2: Cache Update and Data Push When a new data record for an R-101 device is written to Doris, a data synchronization service module captures this data and immediately updates the corresponding Redis entry in the data cache layer. Specifically, this service module updates the hash structure in Redis with the key device:R-101, updating the values ​​of the P-101 and T-101 fields to the latest measurement point values. Simultaneously, the WebSocket service module detects the data updates for P-101 and T-101, queries the subscription relationship table, and finds all client connections subscribed to these two measurement points.

[0081] Step 3: Real-time front-end rendering The WebSocket service module pushes new data points (e.g., {timestamp:1672531201, device:"R-101", point:"P-101", value:1.52}) to all matching clients. Upon receiving this data point, the front-end visualization layer of the central control room's monitoring screen performs incremental rendering using the AntV chart framework. A new data point is added to the real-time graph, and the dashboard pointer smoothly rotates to the latest value of 1.52. The entire process, from data generation to front-end display, has latency controlled to the millisecond level.

[0082] The historical trend analysis process is as follows: Step 1: User Inquiry To troubleshoot a production anomaly, a process engineer needs to view the temperature (T-101) change curve of reactor R-101 over the past 24 hours on the front-end interface. After selecting the corresponding equipment, measuring point, and time range on the front-end page, the engineer clicks the query button.

[0083] Step 2: Cache lookup and origin retrieval The frontend sends a historical data query request to the backend service. The backend service generates a cache key based on the request parameters, for example, query:R-101:T-101:1672444800:1672531200, and queries Redis. Assuming that this time range of data is being queried for the first time, a "cache miss" occurs.

[0084] Step 3: Doris Parallel Computation The backend service translates the query request into an SQL statement and submits it to the Doris cluster. The Doris FE nodes partition and prune the data based on the timestamp, locate the data partition for the past two days, and distribute the query task to the BE nodes that store the R-101 device data. Each BE node scans the data in parallel, completing the query and aggregation of 24 hours (a total of 86,400 data points) within seconds, and returns the result set.

[0085] Step 4: Cache writing and front-end rendering After receiving the result set from Doris, the backend service first writes it to Redis and sets a lifespan of, for example, 600 seconds for the cache key. Then, the backend service sends the result set back to the frontend. The AntV charting framework on the frontend receives the complete dataset containing 86,400 data points, performs a full render, and draws a smooth, complete historical temperature change curve on the interface. Users can analyze the details of the curve through interactive operations such as zooming and panning.

[0086] The experimental verification and effect comparison are as follows: To verify the technical effects of the embodiments of the present invention, the following comparative experimental environment was set up.

[0087] Experimental environment: Hardware configuration: The server is a cluster built using three physical machines with 16-core CPUs and 64GB of memory.

[0088] Data scale: Simulating 1,000 devices, each device reporting 1 measurement point data per second, running continuously for 30 days, the total data volume is approximately 2.6 billion records.

[0089] The system of this invention uses a Doris 1.2 cluster as the data aggregation and storage layer, Redis 6.2 as the data caching layer, and the backend service pushes data via WebSocket.

[0090] Comparison system: It uses the traditional relational database MySQL 8.0 as the data storage layer (with table partitioning optimization). The backend service provides data to the front-end visualization layer via HTTP polling, with a polling cycle of 1 second.

[0091] Historical data query response time comparison: Historical data for a single measurement point across different time spans were queried from both systems, and the response time from initiating the request to receiving all data was recorded. Each query was repeated 10 times, and the average value was taken. The results are shown in Table 1. Table 1. Comparison of Historical Data Query Response Time Comparison of average latency for real-time data updates: The end-to-end average latency from data collection point generation to front-end page rendering was measured, and the performance under different numbers of concurrent clients was statistically analyzed. The results are shown in Table 2: Table 2. Comparison of Average Latency for Real-Time Data Updates As shown in Tables 1 and 2, the response time of this embodiment of the invention is significantly lower than that of the comparative system when processing massive historical data queries. Regarding real-time data presentation, the WebSocket-based push mechanism provides lower update latency compared to HTTP polling, and the latency increase is gradual as the number of clients increases, demonstrating strong system scalability.

[0092] See attached document Figure 3 The test environment consisted of a dataset containing 100 million process data records. The test involved executing historical data queries across three different time spans and recording the response time from initiating the request to receiving the complete data result set.

[0093] like Figure 3 As shown, under the conditions of query time spans of "last 1 hour", "last 1 day", and "last 30 days", the response times of this embodiment are 125ms, 310ms, and 1850ms, respectively. Under the same conditions, the response times of the comparative example are 950ms, 8600ms, and 92400ms, respectively.

[0094] See attached document Figure 4 The test involves recording the average time delay from when a process data point is generated by the backend service to when it is received by the frontend client, under the condition that different numbers of clients connect to and subscribe to real-time data at the same time.

[0095] like Figure 4 As shown, with concurrent client numbers of 10, 100, and 500, the average real-time data update latency in this embodiment of the invention is 75ms, 92ms, and 138ms, respectively. Under the same conditions, the average real-time data update latency in the comparative embodiment is 580ms, 1150ms, and 3200ms, respectively.

[0096] The above test results objectively demonstrate the technical effectiveness of this invention in terms of historical data query response speed and real-time data update latency by integrating distributed storage, data caching, and WebSocket push mechanisms.

Claims

1. A method for presenting massive amounts of process data based on WebSocket, characterized in that, Includes the following steps: Write the process data points generated from industrial data sources into the Doris distributed database cluster; Based on the historical data query request initiated by the front-end visualization layer, a cache key is generated, and the Redis in-memory database is queried based on the cache key; When no data corresponding to the cache key is found in the Redis in-memory database, the Doris distributed database cluster is queried to obtain the query result set. The query result set and the cache key are written into the Redis in-memory database together, and the query result set is sent to the front-end visualization layer. Through the Websocket communication connection established with the front-end visualization layer, a subscription message containing a process measurement point identifier is received; When a new process data point is received, based on the process measurement point identifier contained in the new process data point and the subscription message, the new process data point is pushed to the front-end visualization layer through the Websocket communication connection.

2. The method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The step of writing the process data points into the Doris distributed database cluster further includes: The process data points are stored in a Doris data table that uses the DUPLICATE KEY data model. The Doris data table is configured with a composite sort key consisting of an equipment identifier field, a measurement point identifier field, and a timestamp field. The process data points are distributed and stored across multiple BE nodes in the Doris distributed database cluster by employing a time range partitioning strategy and a hash bucketing strategy based on the device identifier field.

3. The method for presenting massive process data based on WebSocket according to claim 2, characterized in that, The step of writing the process data points into the Doris distributed database cluster is implemented through a Stream Load mechanism, which includes: A data loading request is initiated to the FE node of the Doris distributed database cluster via the HTTP protocol; The FE node designates a coordinating node and sends the data batches encapsulated from the process data points to the coordinating node. The coordinating node distributes the data records in the data batch to the corresponding target BE node for writing based on the equipment identification field of the process data point.

4. The method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The step of writing the query result set and the cache key together into the Redis in-memory database further includes: Set a predetermined lifecycle for the cache key.

5. The method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The step of generating the cache key further includes: The fixed string prefix, the device identifier specified in the historical data query request, the measurement point identifier specified in the historical data query request, the start timestamp specified in the historical data query request, and the end timestamp specified in the historical data query request are concatenated into a unique string, which serves as the cache key.

6. The method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The method further includes: For a single, up-to-date real-time process data point, a Redis hash data structure is used for caching in the Redis in-memory database; For historical trend data sequences within short periods, the Redis SortedSet data structure is used for caching in the Redis in-memory database.

7. The method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The step of pushing the new process data point further includes: Maintain a subscription relationship table that records the mapping relationship between each active WebSocket connection and the set of process measurement point identifiers subscribed to by the connection; When the subscription message is received, the subscription relationship table is updated; When the new process data point is received, the subscription relationship table is queried to retrieve all client connections that have subscribed to the process measurement point identifier of the new process data point, and the push is executed.

8. The method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The method further includes the following steps performed by the front-end visualization layer: Upon receiving the new process data point pushed through the Websocket communication connection, an incremental rendering operation is performed; After receiving the query result set, perform a full rendering operation.

9. The method for presenting massive process data based on WebSocket according to claim 4, characterized in that, The step of setting a predetermined lifespan for the cache key further includes: Set a predefined validity period for the cache key and calculate the absolute expiration timestamp so that the Redis in-memory database automatically deletes the corresponding cache key-value pair when the absolute expiration timestamp is reached.

10. A method for presenting massive process data based on WebSocket according to claim 1, characterized in that, The method further includes an active update mechanism, which includes: An independent background task, running at a preset cycle, actively initiates data queries to the Doris distributed database cluster and writes or overwrites the obtained data results into the corresponding cache key in the Redis in-memory database.

Images