Data stream processing system and method for front-end device, front-end device and storage medium

Abstract

The present application discloses a data stream processing system, method, front-end device and storage medium for a front-end device, which relates to the technical field of server data monitoring. The system can compile user input rules into a binary instruction set module, divide the original data stream of the server into blocks to obtain a binary data stream, use at least one working thread of the data processing engine to call the binary instruction set module, and process the binary data stream in parallel to obtain structured data. Through a visualization component, it is used to generate a service relationship map based on the structured data, and visualize the data service relationship map to obtain an interactive three-dimensional view, eliminating the data transmission and back-end calculation links, without the need for full loading, and realizing real-time rendering of an interactive 3D topology map in the browser, solving technical problems such as data transmission redundancy, high processing delay, low resource utilization and rigid visualization in related technologies, and achieving technical effects such as reducing delay, improving resource utilization and real-time visualization.

Description

Technical Field

[0001] The present application relates to the technical field of server data monitoring, and in particular to a data stream processing system and method for a front-end device, a front-end device, and a storage medium. Background Art

[0002] With the widespread adoption of cloud computing and microservices architectures, the scale of server monitoring data is growing exponentially, posing challenges to traditional backend-centric monitoring architectures. Existing systems centrally perform log collection, parsing, filtering, and aggregation on the backend, while the frontend passively receives structured data for visualization. This results in significant network bandwidth consumption on intermediate data transmission and creates significant redundancy. Furthermore, the multi-level backend processing introduces high latency (average ≥800ms), making it difficult to meet the requirements of real-time monitoring and second-level fault location. This performance issue is particularly prominent in high-frequency, small query scenarios. Furthermore, the backend must maintain a separate computing context for each frontend session, resulting in wasted resources. Most raw data is discarded after processing, leading to low overall computing resource utilization. Furthermore, the visualization component relies on predefined topologies and cannot be dynamically updated, making it difficult to adapt to real-time dependency changes in a microservices architecture. Summary of the Invention

[0003] The present application provides a data stream processing system, method, front-end device and storage medium for a front-end device, so as to at least solve technical problems in related technologies such as data transmission redundancy, high processing delay, low resource utilization and visualization rigidity.

[0004] The present application provides a data stream processing system for a front-end device, wherein the front-end device communicates with a server through a browser, and the browser receives the original data stream of the server. The system includes: a rule compilation engine, which is used to compile user input rules into a binary instruction set module; a data access engine, which receives the original data stream of the server through the browser, and obtains a binary data stream after dividing the original data stream of the server into blocks; a data processing engine, wherein the data processing engine is provided with at least one working thread, and the working thread is loaded with a binary instruction set module, and at least one working thread calls the binary instruction set module and processes the binary data stream in parallel to obtain structured data; a visualization component, which is used to generate a service relationship map based on the structured data, and visually render the service relationship map to obtain an interactive three-dimensional view.

[0005] The present application also provides a front-end device, including the data stream processing system of the above-mentioned front-end device.

[0006] The present application also provides a data stream processing method for a front-end device, wherein the front-end device communicates with a server through a browser, and the browser receives the original data stream of the server. The method utilizes the data stream processing system of the above-mentioned front-end device to process the original data stream of the server, wherein the method includes: executing a rule compilation engine, the rule compilation engine compiles the user input rules into a binary instruction set module; executing a data access engine, the data access engine receives the original data stream of the server through the browser, and obtains a binary data stream after dividing the original data stream of the server into blocks; executing a data processing engine, the data processing engine is provided with at least one working thread, the working thread is loaded with a binary instruction set module, at least one working thread calls the binary instruction set module, and processes the binary data stream in parallel to obtain structured data; executing a visualization component, the visualization component generates a service relationship map based on the structured data, and visually renders the data service relationship map to obtain an interactive three-dimensional view.

[0007] The present application also provides a non-volatile computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the data stream processing method of the front-end device are implemented.

[0008] Through this application, since the user input rules can be compiled into a binary instruction set module, the server's original data stream can be divided into blocks to obtain a binary data stream, the binary instruction set module can be called by at least one working thread of the data processing engine, and the binary data stream can be processed in parallel to obtain structured data. Through the visualization component, a service relationship graph can be generated according to the structured data, and the data service relationship graph can be visualized and rendered to obtain an interactive three-dimensional view, eliminating the data transmission and back-end calculation links, without the need for full loading, and realizing real-time rendering of interactive 3D (Three-Dimensional) topology maps in the browser. Therefore, it can solve the technical problems of related technologies such as data transmission redundancy, high processing delay, low resource utilization and rigid visualization, and achieve technical effects such as reducing delay, improving resource utilization and real-time visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 A schematic diagram of the structure of a data stream processing system of a front-end device provided in an embodiment of the present application;

[0011] Figure 2An architectural diagram of a browser-side server monitoring data stream processing system based on a binary instruction set provided in one embodiment of the present application;

[0012] Figure 3 A diagram showing the working principle of a data access engine provided for one embodiment of the present application;

[0013] Figure 4 A diagram showing the working principle of a memory management unit according to one embodiment of the present application;

[0014] Figure 5 A diagram showing the working principle of a parallel computing unit provided in one embodiment of the present application;

[0015] Figure 6 A diagram showing the working principle of a rule compilation engine provided in one embodiment of the present application;

[0016] Figure 7 A diagram showing the working principle of a rendering engine provided in one embodiment of the present application;

[0017] Figure 8 A flow chart of a data stream processing method of a front-end device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] Related technologies typically employ a backend-centric log processing architecture. Backend services collect raw logs on the server, then parse, filter, aggregate, and store the collected logs. The browser then receives JSON (JavaScript Object Notation) data via HTTP (Hypertext Transfer Protocol) polling or WebSocket (a protocol for full-duplex communication over a single Transmission Control Protocol connection). Charting tools parse the content and render it into relevant charts for display to the user.

[0021] Because related technologies require log parsing, filtering, and aggregation to be performed on the backend, the frontend only receives structured data (such as JSON). This results in 60%-80% of network bandwidth being consumed in intermediate data transmission, leading to redundant frontend-to-backend data transmission. Furthermore, data from these technologies must be processed at multiple levels of the backend (with an average latency of ≥800ms), making them unable to meet real-time monitoring requirements (such as fault location within seconds). Furthermore, high-frequency, small queries (such as dynamic filtering for specific error codes) require repeated backend computation, exacerbating latency. For example, latency can increase by 3-5 times when QPS (Queries Per Second) exceeds 100. Furthermore, these technologies waste resources. The backend cluster must maintain a separate compute context for each frontend session, resulting in excessive memory usage. Over 90% of the raw data is discarded after processing (for example, only 0.1% of abnormal records need to be extracted from the entire log), resulting in less than 15% compute resource utilization. These technologies also suffer from rigid visualization. Topology relationships must be predefined, resulting in high update latency, making them unable to reflect the dynamic dependencies of microservices in real time.

[0022] In response to the defects of the above-mentioned related technologies, this application proposes a data stream processing system, method, front-end device and storage medium for front-end devices to solve technical problems such as data transmission redundancy, high processing delay, low resource utilization and visualization rigidity in related technologies, which will be described in detail below.

[0023] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0024] Figure 1 This is a structural diagram of a data stream processing system of a front-end device according to an embodiment of the present invention. The front-end device communicates with the server through a browser, and the browser receives the original data stream of the server, such as Figure 1 As shown, the data stream processing system 10 of the front-end device includes: a rule compilation engine 101, a data access engine 102, a data processing engine 103 and a visualization component 104.

[0025] Among them, the rule compilation engine 101 is used to compile user input rules into a binary instruction set module; the data access engine 102 receives the original data stream of the server through the browser, and obtains a binary data stream after dividing the original data stream of the server into blocks; the data processing engine 103 is provided with at least one working thread, the working thread is loaded with a binary instruction set module, at least one working thread calls the binary instruction set module, and processes the binary data stream in parallel to obtain structured data; the visualization component 104 is used to generate a service relationship map based on the structured data, and visually render the service relationship map to obtain an interactive three-dimensional view.

[0026] Among them, the binary instruction set module, namely Wasm (WebAssembly), is small in size and loads quickly. It allows server-side code (such as C or C++, two classic programming languages) to be compiled and executed in the browser, and supports a module system, allowing Wasm code to call JavaScript (a programming language widely used for web page interactive behaviors) functions and be called by JavaScript code; the data access engine 102 obtains the original data stream from the server through WebSocket or HTTP / 2 Stream (the data stream of the second major version of the HTTP protocol) protocol.

[0027] It can be understood that the embodiment of the present application includes the following four core components: a rule compilation engine 101, which is used to convert the rules input by the user into a binary instruction set module; a data access engine 102, which obtains the original data stream from the server through the WebSocket or HTTP / 2Stream protocol and divides it into binary data blocks of fixed size; a data processing engine 103, which is configured with at least one working thread, each thread is loaded with a binary instruction set module, and can process these binary data blocks in parallel to generate structured data; a visualization component 104, which generates a service relationship graph based on the structured data, and renders an interactive three-dimensional view based on the generated data service relationship graph, providing an intuitive service dependency display, reducing back-end resource consumption, and improving the real-time and efficiency of data processing.

[0028] In an embodiment of the present application, the rule compilation engine 101 is configured with compilation rules, wherein the compilation rules include: converting the user input rule into an abstract syntax tree; converting the abstract syntax tree into function code; calling the browser's compilation instructions to compile the function code into binary instruction set bytecode; and generating a binary instruction set module based on the binary instruction set bytecode.

[0029] Among them, the Abstract Syntax Tree, or AST, is a tree structure that represents the grammatical structure of the source code. It is an intermediate representation after the source code is parsed, which facilitates further conversion into machine-executable code; the function code is the intermediate code generated from the abstract syntax tree, which is usually a code snippet of a high-level programming language; the binary instruction set bytecode is a low-level, platform-independent instruction set format converted by the compiler into the function code, which is suitable for binary instruction set execution. The compiler can be wasm-pack build, a tool for building and packaging binary instruction set modules, which is responsible for not only compiling the code, but also handling tasks such as generating the correct module format.

[0030] It can be understood that the rule compilation engine 101 of the embodiment of the present application is configured with compilation rules. First, the user input rules are parsed and converted into an abstract syntax tree. Then, specific function codes are generated based on the abstract syntax tree. A high-level programming language such as Rust (a system-level programming language) is usually used to represent these logics; a compiler is used to compile the generated Rust function code into binary instruction set bytecode in a binary instruction set format, and finally, a binary instruction set module that can be efficiently executed in a browser environment is generated based on the binary instruction set bytecode.

[0031] In an embodiment of the present application, the data access engine 102 is configured with a dynamic rate adjustment algorithm, which uses the dynamic rate adjustment algorithm to adjust the data transmission rate between the browser and the server, wherein the dynamic rate adjustment algorithm includes: obtaining the data block size, the browser's memory usage and safety factor; based on the data block size, the browser's memory usage and safety factor, calculating the data transmission rate between the browser and the server.

[0032] Among them, the data block size is pre-configured according to actual needs and is not specifically limited here; the browser's memory usage refers to the ratio of used memory to available memory in the current browser process; the safety factor is a preset value, which is set according to actual needs and is not specifically limited here; the dynamic rate adjustment algorithm is a method for adjusting the data transmission rate according to the current browser's memory usage and the preset safety factor, with the aim of ensuring efficient use of network bandwidth while avoiding performance problems caused by overload. The details will be described in detail below and will not be repeated here.

[0033] It can be understood that the data access engine 102 of the embodiment of the present application is configured with a dynamic rate adjustment algorithm, which is used to adjust the data transmission rate between the browser and the server. Specifically, the dynamic rate adjustment algorithm includes the following steps: first, obtain a pre-configured data block size, and at the same time obtain the browser's memory usage and safety factor; calculate an appropriate data transmission rate based on these parameters, wherein the data block size refers to the size of each data block, the browser's memory usage reflects the amount of memory resources currently available to the browser, and the safety factor is a pre-set value, such as 0.7, which is used to ensure that the system stability can be maintained under high load conditions. Through this algorithm, the data access engine 102 can dynamically adjust the data transmission rate, thereby optimizing the communication efficiency between the browser and the server, and preventing problems such as data processing failure or browser crash due to insufficient memory.

[0034] In the embodiment of the present application, the calculation formula for the data transmission rate is:

[0035]

[0036] in, is the data transmission rate; Remaining memory for the browser; is the data block size; The safety factor is a preset value, which is set according to actual needs. Generally, it can be 0.7, and is not specifically limited here.

[0037] It can be understood that the embodiment of the present application uses this formula, combined with the browser's remaining memory, data block size and safety factor, to dynamically calculate a reasonable data transmission rate, thereby enabling the browser to efficiently and stably receive and process streaming data from the server, optimize the communication efficiency between the browser and the server, and prevent problems such as data processing failure or browser crash due to insufficient memory.

[0038] In an embodiment of the present application, the data access engine 102 is used to divide the original data stream into blocks according to a pre-configured data block size. Each data block includes a timestamp, a server identifier, and original binary data. The processing priority of each data block is determined based on the timestamp, the memory occupied by the processed data blocks is released, and the unprocessed data blocks are stored in the database.

[0039] Among them, the timestamp is the time information contained in each data block, which is used to identify the specific moment when the data block was generated in order to determine its processing priority; the server identifier is the information of the source server contained in each data block, which is used to distinguish different data sources; raw binary data: the unprocessed raw data content received directly from the server; the database can be IndexedDB, a technology for storing large amounts of structured data on the browser side, which can be used to store unprocessed data blocks.

[0040] It can be understood that the data access engine 102 of the embodiment of the present application is also used to block the original data stream according to the pre-configured data block size, wherein each data block includes three parts: a timestamp, a server identifier and original binary data. Specifically, the data access engine 102 first divides the received original data stream according to the preset data block size, such as 4MB, and adds a timestamp and a server identifier to each data block to ensure that its source can be identified and the processing priority can be determined. In the actual processing process, the data access engine 102 will give priority to processing data blocks with newer timestamps to ensure real-time performance. When a data block is processed, the memory it occupies will be immediately released to optimize resource utilization. For data blocks that have not been processed, they are temporarily stored in the database and await subsequent processing. Through the above processing method, not only the data processing efficiency is improved, but also the stability and response speed of the system are ensured.

[0041] In the embodiment of the present application, before the data access engine 102 uses the dynamic rate adjustment algorithm to adjust the data transmission rate between the browser and the server, if it is identified that the memory usage exceeds a pre-configured usage threshold, it stops receiving the original data stream from the server.

[0042] The usage threshold is a pre-set memory usage ratio. When the actual memory usage reaches or exceeds this value, measures will be taken to prevent further memory overflow. The usage threshold is set according to actual needs and is not specifically limited here.

[0043] It can be understood that the data access engine 102 of the embodiment of the present application needs to check the current memory usage before using the dynamic rate adjustment algorithm to adjust the data transmission rate between the browser and the server. If it is identified that the memory usage exceeds the pre-configured usage threshold, the data access engine 102 will immediately stop receiving the original data stream from the server until the memory usage falls back to a safe range. This method can effectively prevent system instability or crashes caused by memory overload, thereby ensuring the stability and efficiency of the system.

[0044] In an embodiment of the present application, the data processing engine 103 includes a memory management unit and a parallel computing unit, wherein the memory management unit maps a shared storage area through an interface, a binary data stream is written to the shared storage area, the structured fields of the binary data stream are extracted from the shared storage area according to the offset, and the data stream whose data size is greater than the target size is stored in the database; the parallel computing unit creates at least one working thread according to the computing resources, each working thread loads a binary instruction set module to parse the structured fields of the binary data stream, and aggregates the structured fields parsed by each working thread to obtain structured data.

[0045] The interface is the WebAssembly.Memory interface, which is a mechanism provided by the binary instruction set for sharing memory between JavaScript and binary instruction set code. The shared storage area is a memory area that allows multiple threads or work units to access it simultaneously (for example, SharedArrayBuffer, a mechanism that allows memory sharing between multiple worker threads (including the main thread and Web Workers), provides a fixed-size buffer that can be directly accessed between different threads, allowing these threads to efficiently read and write the same data without communicating through message passing). The received binary data stream is stored here for efficient reading and processing. The target size is set according to actual needs, for example, 200MB, which is not specifically limited here. The worker thread is also known as the Web Worker, an execution unit dynamically created by the parallel computing unit based on available computing resources. The number of worker threads N = the number of CPU (Central Processing Unit) cores × 2. Each worker thread can independently load and run binary instruction set modules to process data. The database can be IndexedDB.

[0046] It can be understood that the data processing engine 103 of the embodiment of the present application includes two core parts: a memory management unit and a parallel computing unit, wherein the memory management unit writes a shared storage area (such as SharedArrayBuffer) with a binary data stream received from the server through an interface mapping, and the memory management unit can directly extract the structured fields in the binary data stream from the shared storage area according to the offset, and if the size of a data stream exceeds the preset target size, it is stored in the database to avoid occupying too much memory; the parallel computing unit creates at least one working thread according to the computing resource situation of the current system, and each working thread will load a binary instruction set module. These working threads parse the structured fields in the binary data stream assigned to them respectively, and aggregate the parsed results to finally generate a unified structured data output. This method not only improves the speed and efficiency of data processing, but also ensures the stability and responsiveness of the system.

[0047] In an embodiment of the present application, the visualization component 104 includes a topology discovery engine and a rendering engine, wherein the topology discovery engine extracts log feature vectors from structured data, constructs multiple node position relationships of a service relationship graph based on the log feature vectors, calculates the position relationship weights between multiple nodes based on the multiple node position relationships, and updates the multiple node position relationships based on the position relationship weights; the rendering engine renders the service relationship graph based on the multiple node position relationships.

[0048] Among them, the log feature vector is the feature information extracted from the log data, such as the caller IP, target service, response time, error code, etc., which is used to describe the interaction pattern between services; the service relationship graph is an intuitive graphical representation, for example, using multiple nodes to represent different services, and using connecting lines between nodes to represent the dependencies or call relationships between services, helping users to intuitively understand the dependencies between services; the rendering engine is a high-performance rendering engine, such as a high-performance rendering engine based on WebGL (WebGraphics Library, a technology for rendering interactive 3D and 2D graphics in web browsers) technology, which can present the service relationship graph to the user in the form of a 3D view and can support interactive operations such as zooming, rotating and panning.

[0049] It can be understood that the visualization component 104 of the embodiment of the present application includes two main parts: a topology discovery engine and a rendering engine, wherein the topology discovery engine can extract log feature vectors from structured data, including the caller IP, target service, response time, error code, etc., and construct multiple node position relationships in the service relationship graph based on these feature vectors. By further calculating the position relationship weights between each node, the topology discovery engine can dynamically update the position relationship of multiple nodes to ensure that service nodes that frequently interact are close to each other, and nodes that have no direct relationship are far away from each other, forming a clear service dependency relationship graph. Finally, the rendering engine is used to render the service relationship graph based on the above-mentioned updated multiple node position relationships, and present the service relationship graph to the user.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0051] According to the data stream processing system of the front-end device of the embodiment of the present application, the user input rules can be compiled into a binary instruction set module, the original data stream of the server can be divided into blocks to obtain a binary data stream, the binary instruction set module is called by at least one working thread of the data processing engine, and the binary data stream is processed in parallel to obtain structured data. Through the visualization component, a service relationship graph is generated according to the structured data, and the data service relationship graph is visualized and rendered to obtain an interactive three-dimensional view, eliminating the data transmission and back-end calculation links, without the need for full loading, and realizing real-time rendering of interactive 3D topology maps in the browser, achieving technical effects such as reducing latency, improving resource utilization and real-time visualization.

[0052] The data stream processing system of the front-end device is further described below through a specific embodiment.

[0053] This embodiment is a browser-side server monitoring data stream processing system based on a binary instruction set. Through front-end direct processing of the binary instruction set, the data transmission and back-end computing links are eliminated, the delay caused by back-end preprocessing, serialization / deserialization and multi-level transmission in the traditional architecture is eliminated, and the resource consumption of the back-end computing cluster is reduced, especially for high-frequency small query scenarios. Through streaming block processing + IndexedDB database persistence, it supports TB-level data progressive analysis, breaking the limitation of full loading of traditional solutions. Through binary instruction set zero-copy operation + paging swap mechanism, the memory peak is controlled to ≤ 200MB, breaking through the memory and performance bottlenecks of the JavaScript engine, and realizing the safe and stable processing of GB-level or even TB-level raw log streams in the browser. The memory usage is reduced to less than 15% of the traditional solution, avoiding browser crashes caused by excessive data volume. Through real-time compilation from DSL (Domain-Specific Language, domain-specific language) to binary instruction set, the execution speed is close to that of native code, overcoming the JavaScript interpretation and execution of user-defined rules (such as complex SQL (Structured Query Language)). It eliminates performance defects of query language (structured query language) conditions (improved by 5-10 times), supports millisecond-level hot update of rules, and does not require backend participation in compilation and deployment; through unsupervised topology discovery algorithms, it solves the problem of traditional solutions relying on predefined topological relationships, automatically discovers dynamic dependencies between service nodes, and realizes real-time rendering of interactive 3D topology maps in the browser.

[0054] This embodiment is a browser-side server monitoring data stream processing system based on a binary instruction set. It is mainly divided into four core modules: data access engine, data processing engine, rule compilation engine, and visualization component. These modules work together to achieve high-performance data processing and dynamic visualization on the browser side.

[0055] The following is an introduction to the overall system architecture. Figure 2 This is the system architecture diagram of this embodiment, as shown in Figure 2 As shown, this embodiment includes: a rule compilation engine 101, a data access engine 102, a data processing engine 103 and a visualization component 104. Specifically:

[0056] Data Access Engine 102: Receives raw server data streams (such as logs and monitoring metrics) and controls the transmission rate and chunking strategy. It receives binary data streams pushed by the server via WebSocket or HTTP / 2 Stream protocols and then passes the chunked data stream (default 4MB / chunk) to Data Processing Engine 103 through the flow control module.

[0057] Data processing engine 103: Parses, filters, and aggregates raw data to generate structured results. The memory management unit directly manipulates binary data streams using zero-copy technology. The parallel computing framework utilizes Web Workers to implement multi-instance parallel processing.

[0058] Rule Compilation Engine 101: Dynamically compiles user-entered filtering rules (SQL-like, regular expressions) into Wasm modules. For example, the rule expression defined by the user in the front-end interface is dynamically compiled into executable Wasm bytecode and loaded into the worker thread Web Worker.

[0059] Visualization component 104: Generates service topology diagrams in real time and renders interactive 3D views. The topology discovery engine analyzes log features and dynamically calculates node relationships. The 3D rendering engine uses WebGL to achieve high-performance visualization.

[0060] In this embodiment, the overall data flow is: original data flow Data access engine 102 Data processing engine 103 Rule Compilation Engine 101 Structured data Visualization component 104 .

[0061] Each part of this embodiment is described in detail below.

[0062] First is the data access engine 102, Figure 3 The following is a working principle diagram of the data access engine 102, specifically:

[0063] The browser initiates a connection request via WebSocket or HTTP / 2 Stream, negotiates transmission parameters (i.e., data block size), and the server continuously pushes data blocks. Processed blocks are immediately released from memory, while unprocessed blocks are stored in IndexedDB. New blocks are processed first, and historical blocks are loaded during idle time. As blocks are received, the browser dynamically reports the reception rate based on remaining memory.

[0064]

[0065] in, is the data transfer rate (MB / s); The remaining memory of the browser (MB); The data block size (default 4MB); is the safety factor, which takes a value of 0.7.

[0066] The formula for the dynamic rate adjustment algorithm is as above. When the browser memory usage is greater than 80%, the reception of new data blocks is suspended.

[0067] For data chunking, it can be divided into blocks of fixed size. Each data block should contain a timestamp, source server identifier, and original binary data. The latest data block should be processed first.

[0068] The data processing engine 103 is introduced below. The data processing engine 103 includes a memory management unit and a parallel computing unit. Figure 4 This is a diagram of the working principle of the memory management unit, specifically:

[0069] The binary data received by the browser is directly written into SharedArrayBuffer (shared memory area). The Wasm module maps the buffer through the WebAssembly.Memory interface, directly reads the raw bytes, skips the deserialization step during parsing, extracts fields by offset, and automatically pages and stores data exceeding 200MB in the database IndexedDB.

[0070] Figure 5 This is a working principle diagram of the parallel computing unit, specifically:

[0071] The parallel computing unit utilizes Web Workers to implement multi-instance parallel processing. The main thread assigns data blocks to idle Web Workers, dynamically creating Workers based on the number of CPU cores (N = number of cores × 2). Each Worker loads a separate binary instruction set module and performs parsing (extracting structured fields such as HTTP status codes and request paths from the binary stream), filtering (executing user-compiled Wasm rule functions), and statistics (aggregating data that meets the criteria). Each Worker sends the statistical results to the main thread for aggregation, outputting structured results, and feeding them back to the visualization subsystem.

[0072] The following introduces the rule compilation engine 101. Figure 6 This is a working principle diagram of the rule compilation engine 101, specifically:

[0073] First, the SQL-like rules entered by the user are parsed and converted into an abstract syntax tree (AST). The AST is converted into Rust function code, and wasm-pack build is called to generate optimized binary instruction set bytecode. The module is hot-loaded in the Worker through Wasm.instantiateStreaming (a binary instruction set streaming instantiation method that can compile simultaneously during the download process). Finally, a binary instruction set module library is pre-generated for high-frequency rules (such as status code filtering and path matching).

[0074] The visualization component 104 is introduced below. The visualization component 104 includes a topology discovery engine and a rendering engine. Specifically:

[0075] The topology discovery engine is used to analyze log features and dynamically calculate node relationships.

[0076] First, the topology discovery engine performs feature extraction: extracting service call features (caller IP, target service, response time, error code) from the log.

[0077] Next, weight calculation is performed: through the unsupervised topology discovery algorithm, the service relationship graph based on the log feature vector is constructed to build the node relationship strength, and the node and The weight of the relationship between .

[0078]

[0079] in, For nodes To Node The relationship weight of for Call The total number of times; For nodes the number of errors that occurred or the error rate; For nodes delay.

[0080] The topology discovery engine is also used for dynamic layout: a dynamic layout force-directed algorithm is implemented in the WebGL shader to update node positions in real time.

[0081] Repulsion:

[0082] gravitational:

[0083] in, is the repulsive force constant, is the gravitational constant, which is dynamically adjusted according to the canvas size; is the node relationship weight (derived from log analysis); For nodes and nodes The Euclidean distance between them.

[0084] The dynamic layout force-directed algorithm dynamically adjusts the node position and connection layout through physical mechanics simulation, presenting complex service dependencies in an intuitive and efficient way. B) Nodes that are close to each other and have no direct relationship are kept away from each other to avoid visual overlap. Through force-directed balancing, the following goals are achieved:

[0085] Clustering presentation: Microservice groups with high-frequency interactions are automatically clustered.

[0086] Critical path highlighting: High-weight connections (such as core database call links) have thicker lines and brighter colors.

[0087] Abnormal isolation: The faulty node is "pushed away" from the normal cluster due to the enhanced repulsive force, forming a visual isolation zone.

[0088] like Figure 7 The following diagram shows how the rendering engine works, specifically:

[0089] The rendering engine is a 3D rendering engine that implements high-performance visualization based on WebGL. It can draw the implemented dynamic topology based on node attribute data including node relationship strength relationships.

[0090] The node relationship strength is obtained through the topology discovery engine, and different connections are distinguished according to different strengths. The drawing of nodes is based on the following:

[0091]

[0092] in, is the node size; The default size of the base node is 10px; Indicates the CPU utilization (0-100%).

[0093] Node color is mapped according to node health status, and line thickness and line transparency are mapped according to the node relationship strength calculated above.

[0094] Use OffscreenCanvas for rendering. OffscreenCanvas is an interface provided by HTML for rendering graphics in WebWorker.

[0095] Interactive operations can be added: when the mouse hovers over a node, its associated edges and upstream and downstream services are highlighted.

[0096] The following effects are achieved through the visualization component 104:

[0097] No predefined dependencies: completely driven by real-time data.

[0098] Adaptive Complex Networks: Supports highly readable rendering of nodes.

[0099] Performance and real-time performance: GPU (Graphics Processing Unit) acceleration enables efficient browser-side computing.

[0100] Abnormal perception: intuitively mapping service health status through mechanical changes.

[0101] The embodiment of the present application also provides a data stream processing method of a front-end device, Figure 8A flow chart of a data stream processing method for a front-end device provided in an embodiment of the present application; the front-end device communicates with the server through a browser, the browser receives the original data stream of the server, and the method uses the data stream processing system of the front-end device to process the original data stream of the server, such as Figure 8 As shown, the method includes:

[0102] In step S201 , a rule compilation engine is executed, and the rule compilation engine compiles the user input rules into a binary instruction set module.

[0103] It can be understood that the embodiment of the present application first starts the rule compilation engine, which receives the input rules defined by the user on the front end, such as SQL-like expressions or filter conditions in JSON format, and then converts these rules into abstract syntax trees in turn, generates function code, and calls the compiler supported by the browser to compile the function code into binary instruction set bytecode, and finally generates a binary instruction set module that can be efficiently executed in the browser environment.

[0104] In step S202, the data access engine is executed. The data access engine receives the original data stream of the server through the browser, and divides the original data stream of the server into blocks to obtain a binary data stream.

[0105] It can be understood that the embodiment of the present application executes a data access engine, which establishes a connection with the server through the browser, receives the continuously transmitted original data stream, and processes the received original data in blocks, thereby converting it into a binary data stream that is convenient for subsequent processing.

[0106] In step S203, a data processing engine is executed. The data processing engine is provided with at least one working thread. The working thread is loaded with a binary instruction set module. At least one working thread calls the binary instruction set module and processes the binary data stream in parallel to obtain structured data.

[0107] It can be understood that the embodiment of the present application executes the data processing engine and creates at least one worker thread based on the available computing resources. Each worker thread loads the binary instruction set module generated by the rule compilation engine. Subsequently, these worker threads process the binary data stream from the data access engine in parallel by calling the export function in the module, and finally output structured data with a clear structure. This multi-threaded parallel processing mechanism significantly improves the efficiency of data parsing and the overall performance of the system.

[0108] In step S204, a visualization component is executed, which generates a service relationship graph based on the structured data and visually renders the data service relationship graph to obtain an interactive three-dimensional view.

[0109] It can be understood that the embodiment of the present application executes a visualization component, which receives structured data output from the data processing engine, and builds call relationships between services based on this data to generate a service relationship graph. The visualization component uses a rendering engine to visually render the graph and generate a three-dimensional view that can be interacted with in the browser. Users can rotate, zoom and translate the three-dimensional view through mouse or touch operations, so as to more comprehensively observe the dependencies and operating status between services.

[0110] According to the data stream processing method of the front-end device of the embodiment of the present application, the user input rules can be compiled into a binary instruction set module, the original data stream of the server can be divided into blocks to obtain a binary data stream, and the binary instruction set module can be called by at least one working thread of the data processing engine. The binary data stream is processed in parallel to obtain structured data. Through the visualization component, a service relationship graph is generated according to the structured data, and the data service relationship graph is visualized and rendered to obtain an interactive three-dimensional view, eliminating the data transmission and back-end calculation links, without the need for full loading, and realizing real-time rendering of an interactive 3D topology map in the browser, achieving technical effects such as reducing latency, improving resource utilization and real-time visualization.

[0111] For the description of the features in the embodiment corresponding to the data stream processing method of the front-end device, please refer to the relevant description of the embodiment corresponding to the data stream processing system of the front-end device, and no further details will be given here.

[0112] An embodiment of the present application further provides a front-end device, including the data stream processing system of the above-mentioned front-end device.

[0113] An embodiment of the present application further provides a non-volatile computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of the data stream processing method embodiment of the above-mentioned front-end device when running.

[0114] In an exemplary embodiment, the non-volatile computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0115] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] The above is a detailed introduction to the data stream processing system, method, front-end device and storage medium of a front-end device provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data stream processing system for a front-end device, characterized in that: The front-end device communicates with the server through a browser, and the browser receives the original data stream of the server, wherein the system includes: A rule compilation engine, used to compile user input rules into binary instruction set modules; a data access engine, receiving the original data stream from the server through the browser, dividing the original data stream from the server into blocks to obtain a binary data stream, the data access engine being configured with a dynamic rate adjustment algorithm, and utilizing the dynamic rate adjustment algorithm to adjust the data transmission rate between the browser and the server, wherein the dynamic rate adjustment algorithm comprises obtaining a data block size, a memory usage rate of the browser, and a safety factor; and calculating the data transmission rate between the browser and the server based on the data block size, the memory usage rate of the browser, and the safety factor; a data processing engine, the data processing engine being provided with at least one working thread, the working thread being loaded with the binary instruction set module, the at least one working thread calling the binary instruction set module to process the binary data stream in parallel to obtain structured data; A visualization component is used to generate a service relationship graph based on the structured data, and to perform visual rendering on the service relationship graph to obtain an interactive three-dimensional view. The visualization component includes a topology discovery engine and a rendering engine, wherein the topology discovery engine extracts log feature vectors from the structured data, constructs multiple node position relationships of the service relationship graph based on the log feature vectors, calculates position relationship weights between multiple nodes based on the multiple node position relationships, and updates the multiple node position relationships based on the position relationship weights; the rendering engine renders the service relationship graph based on the multiple node position relationships.

2. The data stream processing system of the front-end device according to claim 1, characterized in that: The calculation formula for the data transmission rate is: in, is the data transmission rate; Remaining memory for the browser; is the data block size; is the safety factor.

3. The data stream processing system of the front-end device according to claim 1, characterized in that: Before the data access engine uses the dynamic rate adjustment algorithm to adjust the data transmission rate between the browser and the server, if it is identified that the memory usage exceeds a pre-configured usage threshold, it stops receiving the original data stream from the server.

4. The data stream processing system of the front-end device according to claim 1, characterized in that: The data processing engine includes a memory management unit and a parallel computing unit, wherein: The memory management unit maps a shared storage area through an interface, the shared storage area having the binary data stream written therein, extracts structured fields of the binary data stream from the shared storage area according to the offset, and stores the data stream having a data size larger than the target size in a database; The parallel computing unit creates at least one working thread according to computing resources, and each working thread loads the binary instruction set module to parse the structured fields of the binary data stream, and aggregates the structured fields parsed by each working thread to obtain the structured data.

5. The data stream processing system of the front-end device according to claim 1, characterized in that: The rule compilation engine is configured with compilation rules, wherein the compilation rules include: Convert user input rules into an abstract syntax tree; Converting the abstract syntax tree into function code; Calling the browser's compilation instruction to compile the function code into binary instruction set bytecode; A binary instruction set module is generated according to the binary instruction set bytecode.

6. A front-end device, characterized in that: A data stream processing system comprising the front-end device according to any one of claims 1 to 5.

7. A data stream processing method for a front-end device, characterized in that: The front-end device communicates with the server via a browser, the browser receives the original data stream from the server, and the method processes the original data stream from the server using the data stream processing system of the front-end device according to any one of claims 1 to 5, wherein the method includes: Executing a rule compilation engine, wherein the rule compilation engine compiles the user input rules into a binary instruction set module; executing a data access engine, wherein the data access engine receives an original data stream from the server through the browser, blocks the original data stream from the server to obtain a binary data stream, and the data access engine is configured with a dynamic rate adjustment algorithm, and uses the dynamic rate adjustment algorithm to adjust the data transmission rate between the browser and the server, wherein the dynamic rate adjustment algorithm includes obtaining a data block size, a memory usage rate of the browser, and a safety factor; and calculating the data transmission rate between the browser and the server based on the data block size, the memory usage rate of the browser, and the safety factor; executing a data processing engine, wherein the data processing engine is provided with at least one working thread, the working thread is loaded with the binary instruction set module, and the at least one working thread calls the binary instruction set module to process the binary data stream in parallel to obtain structured data; Execute a visualization component, which generates a service relationship graph based on the structured data, and visually renders the service relationship graph to obtain an interactive three-dimensional view. The visualization component includes a topology discovery engine and a rendering engine, wherein the topology discovery engine extracts log feature vectors from the structured data, constructs multiple node position relationships of the service relationship graph based on the log feature vectors, calculates position relationship weights between multiple nodes based on the multiple node position relationships, and updates the multiple node position relationships based on the position relationship weights; the rendering engine renders the multiple node position relationships to obtain a service relationship graph.

8. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, wherein the computer program, when executed by the processor, implements the steps of the data stream processing method of the front-end device as claimed in claim 7.

Images