Multi-type XML device data processing method and device based on Netty

Abstract

The invention discloses a multi-type XML (Extensible Markup Language) equipment data processing method and device based on Netty. The method comprises the following steps: constructing a TCP long connection server through a Netty framework to receive a device coding data message; decoding and decompressing the message to obtain XML format data; analyzing the XML data to extract a type identification field; routing the data to a corresponding service processing strategy according to the type identification field; packaging the data into a processing task, and delivering the processing task to a corresponding thread pool for asynchronous processing based on the priority; and finally, converting the processed data into a unified data structure and storing the unified data structure in a database. High-concurrency receiving and processing of equipment data are achieved, multi-type XML message analysis is supported through dynamic routing, key data processing timeliness is guaranteed based on priority scheduling, the data processing throughput is improved by adopting an asynchronous mechanism, and high-concurrency receiving and processing of equipment data are achieved through a unified data conversion standard storage structure. And the efficiency, the real-time performance and the expandability of data processing are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method and apparatus for processing multi-type XML device data based on Netty. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and industrial automation technologies, the number of terminal devices connected to networks is growing exponentially. Various sensors, monitoring and control devices, and smart gateways need to transmit collected data to a central server in real time with extremely high frequency and extremely low latency to support core business functions such as remote monitoring, intelligent decision-making, data analysis, and fault early warning. Against this backdrop, communication protocols between devices and servers are gradually becoming standardized and lightweight. Among these, data message formats based on Extensible Markup Language (XML) are widely adopted by the industry due to their clear structure, high readability, and excellent cross-platform capabilities.

[0003] On the other hand, the server faces enormous challenges in handling massive concurrent device connections, real-time processing of large amounts of data, and parsing diverse and heterogeneous data. Traditional server architectures typically employ blocking I / O or multi-threaded models to manage network connections and data requests. However, these architectures reveal significant performance bottlenecks and design flaws when faced with scenarios involving tens of thousands of devices connecting simultaneously and frequently reporting different types of XML data packets. Rigid protocol processing: Most existing systems are customized for specific device models or single data formats, and their message processing logic is highly coupled with business code. When it is necessary to access new devices or process XML messages with different structures, the core code must be modified and the system redeployed. The lack of a unified and scalable message recognition and routing mechanism leads to poor system adaptability and high maintenance costs.

[0004] Lack of resource scheduling mechanism: Different types of messages (such as emergency alarms, device heartbeats, and historical data uploads) have drastically different real-time requirements. However, traditional systems generally lack a task scheduling mechanism based on business priority. All messages are usually placed into the same queue for processing without distinction, which can lead to high-priority emergency alarm messages being blocked by a large number of low-priority historical data messages, making it impossible to guarantee the real-time response of critical business and posing a huge operational risk.

[0005] Synchronous processing performance bottleneck: In the data processing stage, traditional systems mostly adopt a synchronous blocking processing method. That is, the server can only start processing the next message after it has completed the complete parsing, business processing, and database write operation of a message. This "request-response" serial processing model severely limits the system's throughput and concurrent processing capabilities, cannot fully utilize the computing resources of modern multi-core servers, and easily becomes a bottleneck for system performance during data surges.

[0006] The system modules are tightly coupled: steps such as data decryption, decompression, format parsing, business logic, and data persistence are often intertwined in a single, lengthy processing flow. This tight coupling structure makes the system difficult to maintain, test, and extend; any modification to any part can trigger unforeseen side effects, increasing the system's complexity and technical debt.

[0007] Therefore, how to achieve high-performance, highly scalable, intelligently scheduled, and modularly decoupled device data reception and processing is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0008] This invention provides a method and apparatus for processing multi-type XML device data based on Netty, which can improve data processing throughput, real-time performance and scalability.

[0009] The first aspect of this invention provides a method for processing multi-type XML device data based on Netty, comprising: A TCP long-connection server built using the Netty framework receives encoded data packets from the device. The encoded data message is decoded and decompressed to obtain an XML format data message; Parse XML format data messages and extract the type identifier field; Based on the type identifier field, the XML formatted data message is routed to the corresponding business processing strategy; The XML format data message is encapsulated into a processing task, and the processing task is delivered to the corresponding thread pool for asynchronous processing based on the priority of the business processing strategy. The processed data is converted into a unified data structure and stored in the database.

[0010] Optionally, the encoded data message is decoded and decompressed to obtain an XML-formatted data message, including: The encoded data packets are decoded and restored to a compressed data stream, wherein the encoded data packets are encoded using a 64-character encoding method; The compressed data stream is decompressed using a lossless data compression algorithm to obtain an XML-formatted data packet.

[0011] Optionally, data packets are routed to the corresponding business processing strategy based on the type identifier field, including: The message type is determined by matching a predefined enumeration class based on the type identifier field. Based on the determined message type, a corresponding business processing strategy instance is generated using the factory pattern. The data packet and the generated business processing policy instance are submitted to the policy execution context to execute the business logic encapsulated in the business processing policy instance.

[0012] Optionally, based on the priority of the business processing strategy, the processing tasks are submitted to the corresponding thread pool for asynchronous processing, including: The processing tasks are divided into three priority levels: high, medium, and low. High-priority tasks are processed using a high-priority thread pool, medium-priority tasks are processed using a medium-priority thread pool, and low-priority tasks are processed using a low-priority thread pool. Each thread pool is configured with independent queue capacity, number of threads, and rejection policy.

[0013] Optionally, the method also includes: Configure different maximum number of retries and retry intervals for processing tasks with different priorities; Whether to trigger a retry depends on the type of exception, which includes network exceptions, I / O exceptions, and business logic exceptions. Retry is triggered when the exception type is network exception or IO exception; no retry is triggered when the exception type is business logic exception.

[0014] Optionally, the processed data is converted into a unified data structure and stored in a database, including: XML messages are converted into standard data structures using a unified mapping model to adapt to database storage formats; Encapsulate standard data structures into messages and send them to a message middleware for buffering; Messages are asynchronously retrieved from the message middleware through an independent consumption service, database insertion operations are performed, and reliable data storage is ensured based on a message acknowledgment mechanism.

[0015] Optionally, the method also includes: Based on the data stored in the database, generate device status information, operating indicators, and alarm information for visualization. The device status information, operating indicators, and alarm information are provided to the front-end visualization system for display and monitoring in the form of charts.

[0016] A second aspect of the present invention provides a Netty-based multi-type XML device data processing apparatus, comprising: The communication access and decoding module is used to receive encoded data packets from the device through a TCP long connection server built with the Netty framework; and to decode and decompress the encoded data packets to obtain data packets in XML format. The data type identification and dynamic routing module is used to parse XML format data messages, extract the type identifier field, and route the XML format data messages to the corresponding business processing strategies based on the type identifier field. The multi-level task scheduling module is used to encapsulate XML format data messages into processing tasks and, based on the priority of the business processing strategy, deliver the processing tasks to the corresponding thread pool for asynchronous processing. The data abstraction and persistence module is used to transform processed data into a unified data structure and store it in the database.

[0017] A third aspect of the present invention provides a multi-type XML device data processing device based on Netty, comprising: One or more processors; A memory on which one or more programs are stored; When one or more programs are executed by one or more processors, the one or more processors implement a Netty-based method for processing multi-type XML device data, as described above.

[0018] A fourth aspect of the present invention provides a computer storage medium for storing a program, which, when executed, is used to implement a Netty-based method for processing multi-type XML device data as described in any of the above.

[0019] This invention discloses a method and apparatus for processing multi-type XML device data based on Netty. The method includes: constructing a TCP long-connection server using the Netty framework to receive device-encoded data packets; decoding and decompressing the packets to obtain XML format data; parsing the XML data to extract a type identifier field; routing the data to the corresponding business processing strategy based on the type identifier field; encapsulating the data into processing tasks and delivering them to the corresponding thread pool for asynchronous processing based on priority; and finally converting the processed data into a unified data structure and storing it in a database. This invention achieves high-concurrency reception and processing of device data, supports multi-type XML packet parsing through dynamic routing, ensures the timeliness of critical data processing based on priority scheduling, improves data processing throughput through an asynchronous mechanism, and significantly improves data processing efficiency, real-time performance, and scalability through a unified data conversion and standardized storage structure. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for processing multi-type XML device data based on Netty, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the Netty communication model provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a multi-type XML device data processing device based on Netty provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a multi-type XML device data processing device based on Netty, provided in an embodiment of the present invention. Detailed Implementation

[0022] This invention provides a method and apparatus for processing multi-type XML device data based on Netty, which can improve data processing throughput, real-time performance and scalability.

[0023] See Figure 1 This figure is a flowchart illustrating a method for processing multi-type XML device data based on Netty, provided by an embodiment of the present invention. The method for processing multi-type XML device data based on Netty provided by this embodiment of the present invention can be implemented, for example, through the following steps S101-106.

[0024] S101: A TCP long-connection server built using the Netty framework receives encoded data packets from the device.

[0025] In this embodiment of the invention, a TCP long-connection server is built based on the Netty framework, and a custom ChannelHandler chain is configured.

[0026] S102: Decode and decompress the encoded data message to obtain an XML format data message.

[0027] In this embodiment of the invention, the encoded data packet is decoded to restore it to a compressed data stream, wherein the encoded data packet is encoded using a 64-character encoding method; the compressed data stream is decompressed using a lossless data compression algorithm to obtain an XML format data packet.

[0028] Specifically, the received Base64 encoded message is decoded and further decompressed (supporting Zlib, GZIP, etc.) to obtain a data message in standard XML format.

[0029] S103: Parse the XML format data message and extract the type identifier field.

[0030] In this embodiment of the invention, type identifier fields in XML are extracted using XPath or DOM / SAX parsing. Specifically, this invention employs an adaptive type recognition algorithm based on XPath expressions, which dynamically parses the XML message structure through a pre-defined configuration table or rule engine. By selecting an appropriate XPath path expression based on the device model or service category, the type identifier field can still be accurately extracted even when the XML message structure changes, significantly improving the adaptability and stability of the parsing algorithm.

[0031] S104: Route XML formatted data messages to the corresponding business processing strategy based on the type identifier field.

[0032] In this embodiment of the invention, the message type is determined by matching a predefined enumeration class according to the type identifier field; based on the determined message type, a corresponding business processing strategy instance is generated through the factory pattern; the data message and the generated business processing strategy instance are submitted to the strategy execution context to execute the business logic encapsulated in the business processing strategy instance.

[0033] Specifically, the dynamic routing algorithm of this invention adopts a three-layer structure of enumeration-factory-policy: all recognizable XML message types are uniformly defined through the enumeration class TagNameEnum to ensure system type security; the mapping relationship between message types and business processing classes is established through the factory pattern to achieve scalable and maintainable task scheduling; the processing logic of each type of message is encapsulated through the policy pattern, so that the system can dynamically load and execute the corresponding processing policy according to the enumeration type at runtime; compared with the traditional hard-coded switch-case structure, this invention realizes dynamic type recognition, automated task distribution, and modular system expansion, significantly improving processing performance and system maintainability.

[0034] S105: Encapsulate XML-formatted data messages into processing tasks, and deliver the processing tasks to the corresponding thread pool for asynchronous processing based on the priority of the business processing strategy.

[0035] In this embodiment of the invention, the processing tasks are divided into three priority levels: high, medium, and low. High-priority tasks are processed using a high-priority thread pool, medium-priority tasks are processed using a medium-priority thread pool, and low-priority tasks are processed using a low-priority thread pool. Each thread pool is configured with independent queue capacity, number of threads, and rejection policy.

[0036] Specifically, this invention classifies received XML device messages into three priority levels—high, medium, and low—based on task characteristics: High priority (events, alarms, device logins): Small message size, high real-time requirements, must be processed immediately; Medium priority (heartbeat, status reporting): Periodic tasks with moderate requirements for response latency; Low priority (historical data uploads such as upload, bmsupload, emsupload, meterupload, etc.): high message frequency and large data volume, but can tolerate a certain delay.

[0037] For each type of task, an independent thread pool and queue are configured: a high-pri-exec thread pool with 8 core threads, a maximum of 16 threads, and a queue capacity of 1000, using CallerRunsPolicy to ensure that critical tasks are executed immediately; a medium-pri-exec thread pool with 4 core threads, a maximum of 8 threads, and a queue capacity of 2000, using DiscardOldestPolicy to ensure stable processing of periodic tasks; and a low-pri-exec thread pool with 2 core threads, a maximum of 4 threads, and a queue capacity of 5000, using DiscardOldestPolicy to achieve buffered execution and high throughput for high-frequency tasks.

[0038] This invention parses XML messages using XPath or DOM after receiving them. <type>The system matches the corresponding TagNameEnum field and calls calculatePriority(type) to determine the task priority (0 for high priority, 1 for medium priority, and 2 for low priority). Based on the task priority, the system asynchronously submits the task to the corresponding thread pool queue, where it is consumed independently by a thread. After execution, the thread is automatically recycled without blocking new tasks. Through priority queues, thread pool isolation, and asynchronous scheduling strategies, this invention achieves immediate response for high-priority tasks, stable periodic processing for medium-priority tasks, and high-throughput buffering and database storage for low-priority tasks, significantly improving the real-time performance, throughput, and system stability of asynchronous data processing.

[0039] In one implementation of this invention, different maximum number of retries and retry intervals are configured for processing tasks of different priorities; whether to trigger a retry is determined based on the exception type, wherein the exception type includes network exception, IO exception and business logic exception; when the exception type is network exception or IO exception, a retry is triggered; when the exception type is business logic exception, a retry is not triggered.

[0040] Specifically, this invention designs a task failure retry mechanism, setting a maximum number of retries. For example, high-priority tasks are retried 2-3 times to ensure successful execution of critical tasks; medium-priority tasks are retried 1-2 times to stably complete periodic tasks; and low-priority tasks are retried 1 time to process high-frequency historical data tasks. High-priority tasks use a fixed or slightly exponential backoff method with short intervals, such as 200-500ms, to achieve rapid retries; the intervals for medium and low-priority tasks are gradually increased to, for example, 1-2 seconds to reduce system pressure.

[0041] Exception handling is categorized as follows: network or I / O exceptions (such as database timeouts or Redis connection errors) trigger retries; business logic exceptions (such as message parsing errors or format errors) are not retried but logged; when the thread pool is saturated or the queue overflows, high-priority tasks trigger rapid retries or short-term blocking, while low-priority tasks discard the oldest task. Batch processing optimization (low-priority tasks) includes processing high-frequency messages in batches to reduce context switching overhead and improve throughput; large queue capacity absorbs sudden surges in reported data and maintains system stability.

[0042] Through the above scheduling and retry mechanism, this invention realizes a comprehensive scheduling strategy that enables immediate response for high-priority tasks, stable periodic processing for medium-priority tasks, and high-throughput buffering for low-priority tasks, thereby improving the real-time performance, reliability, and system throughput of asynchronous data processing.

[0043] It should be noted that the embodiments of the present invention do not impose specific limitations on the setting of the maximum number of retries and the retry interval time. The above are just examples, and the specific settings should be made according to actual needs.

[0044] S106: Convert the processed data into a unified data structure and store it in the database.

[0045] In this embodiment of the invention, XML messages are converted into standard data structures through a unified mapping model to adapt to the database storage format; the standard data structures are encapsulated into messages and sent to the message middleware for buffering; messages are asynchronously obtained from the message middleware through an independent consumption service, database insertion operations are performed, and reliable data storage is ensured based on a message acknowledgment mechanism.

[0046] Specifically, various XML messages are transformed into standard data structures through a unified mapping model (DTO); structured data is written to the database for storage through a unified persistence service interface; the database storage supports multiple databases (such as MySQL and PostgreSQL) and asynchronous buffers.

[0047] The Unified Mapping Model (UMM) of this invention adopts a layered structure design for standardized abstraction and unified modeling of various types of XML messages. The model consists of two parts: a core data structure layer and a mapping configuration layer.

[0048] The core data structure layer carries the parsed, standardized message data. This structure uniformly defines fields such as device identifier (deviceId), message type (msgType), reception time (recvTime), parsing time (parseTime), data body (payload), and metadata (metaInfo). Through this structure, XML messages from different devices and in different formats can be transformed into a consistent data representation, facilitating subsequent data storage and analysis.

[0049] The mapping configuration layer is used to define the correspondence between XML nodes and standard fields. The system automatically loads the corresponding mapping configuration file according to the message type, and maps the XML node content to a unified data object through the parsing engine, thereby realizing automatic adaptation and standardized conversion of different message types.

[0050] This design enables unified parsing and structured storage of data from multiple device types, allowing the system to quickly expand to new message types through configuration, reducing code modifications and improving the system's versatility, scalability, and data analysis efficiency.

[0051] This invention employs an asynchronous buffering mechanism in the data persistence stage, using RabbitMQ as the core implementation module. After completing XML message parsing and standardized mapping, the system sends the structured data object as a message to the designated RabbitMQ exchange. The consumer asynchronously receives messages from the corresponding queue and performs database insertion operations, decoupling data reception and storage. To ensure data consistency and reliability, RabbitMQ is configured in persistence mode, and all messages are written to disk before being enqueued. The consumer uses an ACK confirmation mechanism, returning an acknowledgment signal only after successful data insertion to prevent message loss. In case of insertion failure, the system automatically redelivers the message using a NACK and retry mechanism, and uses a dead-letter queue to isolate abnormal data, thereby ensuring the reliability of message transmission and eventual data consistency.

[0052] In one implementation of this invention, based on data stored in a database, device status information, operating indicators, and alarm information for visualization are generated; the device status information, operating indicators, and alarm information are provided to a front-end visualization system for display and monitoring in the form of charts.

[0053] Specifically, all processing stages have log collection and monitoring capabilities, employing a lightweight log collection and monitoring mechanism. Logs from each processing stage are uniformly output to local files, and runtime status is tracked using the RabbitMQ console and custom monitoring tasks. Structured logs are output using a built-in logging framework (such as Logback), supporting hierarchical recording and rolling storage by date. When anomalies occur (such as parsing failures, data insertion errors, etc.), alarm information is pushed via email or message queues. Scheduled tasks monitor RabbitMQ queue length and consumption rate; if task backlog or the number of processing failures exceeds a threshold, alarm logs are automatically generated and maintenance personnel are notified.

[0054] The persisted data is then used by subsequent data visualization systems (such as Grafana and ECharts) for display. The visualization interface of this invention is used to display and monitor the collected and persisted device data, providing intuitive visualization support for operation and maintenance and data analysis. Through data integration with the front-end visualization system, it enables graphical display of device operating status and historical data. The main functions include: Device overview function: Real-time statistics and display of the total number and category distribution of all connected devices; Online status monitoring: Displays the real-time online and offline status of each device in the form of a list or chart, and supports status refresh and alarm flags; Energy Storage Data Overview: Displays key operating indicators such as power, energy consumption, voltage, and temperature of each energy storage system; Alarm and fault status display: Summarizes and categorizes current and historical alarm information, and supports filtering by time, device or alarm level; Historical data analysis: Displays historical trend changes in chart form, including time series analysis of indicators such as power, temperature, and alarm frequency.

[0055] It supports various visualization chart types, including pie charts, line charts, bar charts, and data trend comparison charts. Users can customize the chart display according to their monitoring needs. Users can monitor the overall system operation status in real time, quickly locate faulty equipment, and perform energy efficiency analysis and historical trend assessment, thereby improving the system's operation and maintenance efficiency and data utilization value.

[0056] This invention comprehensively optimizes the receiving, decoding, parsing, and persistence process of XML data packets for various types of devices, and has the following advantages: High-concurrency performance assurance: Based on Netty's non-blocking I / O and event-driven mechanism, it supports tens of thousands of concurrent device connections. This invention employs a Netty-based asynchronous non-blocking I / O model combined with a message queue buffering mechanism (such as RabbitMQ) and multi-threaded task scheduling to decouple and asynchronously process data reception, parsing, and database persistence, thus maintaining stable performance under high-concurrency environments. The system ensures real-time response and reliable database persistence of critical messages through thread pool traffic distribution, high-priority task queues, and message persistence strategies. Furthermore, this invention supports horizontal scaling and cluster deployment, allowing for the addition of server nodes based on device access volume and business load, thereby expanding processing capacity and enabling large-scale device access and high-concurrency data processing. Although this invention does not provide actual stress test data, this technical architecture demonstrates the feasibility of maintaining stable and efficient data processing under large-scale device access conditions.

[0057] The data structure is highly scalable: a unified XML data structure parsing and routing mechanism supports the rapid integration of new device types and data formats.

[0058] Multi-level priority scheduling ensures the real-time performance of critical business operations: Through thread pool priority control, high-priority messages receive faster processing responses, ensuring business timeliness.

[0059] Highly modular and decoupled: The processing flow of decoding, decompression, parsing, routing, and database entry is highly decoupled, making it easy to maintain and expand.

[0060] Supports subsequent data visualization and analysis: The persisted structured data is directly connected to the reporting system, improving data transparency and visualization effects.

[0061] Now combined Figure 2 For example, the core of this invention is to build a modular asynchronous data processing pipeline based on the Netty high-performance network framework.

[0062] First, initialize the Netty server. Its communication model is shown in the attached diagram. Figure 2 As shown, the core components include: BossGroup: As the main thread group, it is responsible for listening for TCP connection requests. When a new device initiates a connection (see attached diagram), it will respond. Figure 1 In step 1. TCP connection request, BossGroup will accept this connection.

[0063] WorkerGroup: As a group of worker threads, it is responsible for handling I / O operations of established connections. After accepting a connection (step 2. Assign connection), BossGroup will register this connection (Channel) to a thread in the WorkerGroup (step 3. Register Channel).

[0064] ChannelPipeline: Each established connection has a corresponding ChannelPipeline, which is essentially a chain of responsibility containing a series of ordered ChannelHandlers.

[0065] Secondly, there is the data processing flow. When a device sends data through an established long-connection communication channel (step 4. Sending data frames), the data will flow through each Handler in the ChannelPipeline: Decoder: First, data enters the decoder unit. In this invention, this includes custom Handlers that execute sequentially: Decode the received data packets that use a 64-character encoding method.

[0066] The decoded data is decompressed using lossless data compression algorithms such as Zlib or GZIP to restore the standard XML format data message.

[0067] Convert the byte stream to a string (Step 5. Byte Decoding).

[0068] BusinessHandler: After obtaining the XML string, the process enters the core business logic unit (step 6. Processing business logic). This unit performs the following operations: Parsing and Type Recognition: Extracting XML data using XPath expressions or DOM / SAX parsers <type>The label serves as a type identifier field.

[0069] Dynamic routing: Based on the type identifier field, it matches a predefined enumeration class to determine the message type, and then generates the corresponding business processing strategy instance through the factory pattern.

[0070] Task scheduling and execution: Determine task priorities and asynchronously submit the encapsulated processing tasks to the corresponding priority thread pools (high, medium, and low) for execution. The logic encapsulated in the business processing strategy instance is executed here.

[0071] Encoder: If a response is required to the device, the response data generated by the business processor will be encoded by the encoder (step 7. Generate Response) and then sent back to the device over the network.

[0072] Finally, there's data abstraction and persistence. After the business processing logic is executed, the generated data is transformed into a standard data structure through a unified mapping model. This structured data is then encapsulated as messages and sent to message middleware such as RabbitMQ for buffering. Ultimately, an independent consumer service asynchronously retrieves the messages from the message queue and performs database insertion operations, ensuring reliable data storage through a message acknowledgment mechanism.

[0073] Through the above implementation methods, the present invention constructs a complete, efficient, and decoupled data processing pipeline from connection management, data decoding, intelligent parsing, dynamic routing, priority scheduling to reliable storage, which significantly improves the concurrency, real-time performance, and maintainability of device data processing.

[0074] Based on the methods provided in the above embodiments, this invention also provides a Netty-based multi-type XML device data processing device, which is described below with reference to the accompanying drawings.

[0075] See Figure 3 This figure is a schematic diagram of the structure of a multi-type XML device data processing device based on Netty provided in an embodiment of the present invention.

[0076] The Netty-based multi-type XML device data processing apparatus 300 provided in this embodiment of the invention includes: a communication access and decoding module 301, a data type identification and dynamic routing module 302, a multi-level task scheduling module 303, and a data abstraction and persistence module 304.

[0077] The communication access and decoding module 301 is used to receive encoded data packets from the device through a TCP long connection server built with the Netty framework; and to decode and decompress the encoded data packets to obtain data packets in XML format. The data type identification and dynamic routing module 302 is used to parse XML format data messages, extract the type identifier field, and route the XML format data messages to the corresponding business processing strategies based on the type identifier field. The multi-level task scheduling module 303 is used to encapsulate XML format data messages into processing tasks and, based on the priority of the business processing strategy, deliver the processing tasks to the corresponding thread pool for asynchronous processing. The data abstraction and persistence module 304 is used to convert the processed data into a unified data structure and store it in the database.

[0078] In one possible implementation, the communication access and decoding module 301 is specifically used for: The encoded data packets are decoded and restored to a compressed data stream, wherein the encoded data packets are encoded using a 64-character encoding method; The compressed data stream is decompressed using a lossless data compression algorithm to obtain an XML-formatted data packet.

[0079] In one possible implementation, the data type identification and dynamic routing module 302 is specifically used for: The message type is determined by matching a predefined enumeration class based on the type identifier field. Based on the determined message type, a corresponding business processing strategy instance is generated using the factory pattern. The data packet and the generated business processing policy instance are submitted to the policy execution context to execute the business logic encapsulated in the business processing policy instance.

[0080] In one possible implementation, the multi-level task scheduling module 303 has the following functions: The processing tasks are divided into three priority levels: high, medium, and low. High-priority tasks are processed using a high-priority thread pool, medium-priority tasks are processed using a medium-priority thread pool, and low-priority tasks are processed using a low-priority thread pool. Each thread pool is configured with independent queue capacity, number of threads, and rejection policy.

[0081] One possible implementation also includes a task failure retry module, specifically used for: Configure different maximum number of retries and retry intervals for processing tasks with different priorities; Whether to trigger a retry depends on the type of exception, which includes network exceptions, I / O exceptions, and business logic exceptions. Retry is triggered when the exception type is network exception or IO exception; no retry is triggered when the exception type is business logic exception.

[0082] In one possible implementation, the data abstraction and persistence module 304 has features for: XML messages are converted into standard data structures using a unified mapping model to adapt to database storage formats; Encapsulate standard data structures into messages and send them to a message middleware for buffering; Messages are asynchronously retrieved from the message middleware through an independent consumption service, database insertion operations are performed, and reliable data storage is ensured based on a message acknowledgment mechanism.

[0083] One possible implementation also includes a monitoring and visualization interface module, specifically used for: Based on the data stored in the database, generate device status information, operating indicators, and alarm information for visualization. The device status information, operating indicators, and alarm information are provided to the front-end visualization system for display and monitoring in the form of charts.

[0084] Since the device 300 is a device corresponding to the Netty-based multi-type XML device data processing method provided in the above method embodiments, the specific implementation of each module of the device 300 is based on the same concept as the above method embodiments. Therefore, for the specific implementation of each module of the device 300, please refer to the description of the Netty-based multi-type XML device data processing method in the above method embodiments, and it will not be repeated here.

[0085] This invention also provides a multi-type XML device data processing device based on Netty, the device comprising: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory and perform the Netty-based multi-type XML device data processing method mentioned in the above embodiments.

[0086] It should be noted that the hardware structure of the multi-type XML device data processing device based on Netty provided in the embodiments of the present invention can be as follows: Figure 4 The structure shown, Figure 4 This is a schematic diagram of the structure of a multi-type XML device data processing device based on Netty, provided in an embodiment of the present invention.

[0087] Please see Figure 4 As shown, device 400 includes: processor 410, communication interface 420, and memory 430. The number of processors 410 in device 400 can be one or more. Figure 4 Taking a processor as an example, in this embodiment of the invention, the processor 410, communication interface 420, and memory 430 can be connected via a bus system or other means. Figure 4 Taking the connection between China and Israel via the bus system 440 as an example.

[0088] Processor 410 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 410 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0089] The memory 430 may include volatile memory, such as random-access memory (RAM); the memory 430 may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 430 may also include a combination of the above types of memory.

[0090] Optionally, the memory 430 stores an operating system and programs, executable modules, or data structures, or subsets thereof, or extended sets thereof. The programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and handling hardware-based tasks. The processor 410 can read the programs from the memory 430 to implement the Netty-based multi-type XML device data processing method provided in this embodiment of the invention.

[0091] The bus system 440 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system 440 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0092] This invention also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the Netty-based multi-type XML device data processing method mentioned in the above embodiments.

[0093] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the Netty-based multi-type XML device data processing method mentioned in the above embodiments.

[0094] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.< / type> < / type>

Claims

1. A method for processing multi-type XML device data based on Netty, characterized in that, The method includes: A TCP long-connection server built using the Netty framework receives encoded data packets from the device. The encoded data message is decoded and decompressed to obtain an XML format data message; Parse the XML format data message and extract the type identifier field; The XML formatted data message is routed to the corresponding business processing strategy based on the type identifier field. The XML format data message is encapsulated into a processing task, and the processing task is delivered to the corresponding thread pool for asynchronous processing based on the priority of the business processing strategy. The processed data is converted into a unified data structure and stored in the database.

2. The method according to claim 1, characterized in that, The process of decoding and decompressing the encoded data packet to obtain an XML-formatted data packet includes: The encoded data packet is decoded to restore it to a compressed data stream, wherein the encoded data packet is encoded using a 64-character encoding method; The compressed data stream is decompressed using a lossless data compression algorithm to obtain the XML format data packet.

3. The method according to claim 1, characterized in that, The step of routing the data packet to the corresponding business processing strategy according to the type identifier field includes: Based on the type identifier field, a predefined enumeration class is matched to determine the message type; Based on the determined message type, a corresponding business processing strategy instance is generated using the factory pattern. The data packet and the generated business processing policy instance are submitted to the policy execution context to execute the business logic encapsulated in the business processing policy instance.

4. The method according to claim 1, characterized in that, The step of submitting processing tasks to the corresponding thread pool for asynchronous processing based on the priority of the business processing strategy includes: The processing tasks are divided into three priority levels: high, medium, and low. High-priority tasks are processed using a high-priority thread pool, medium-priority tasks are processed using a medium-priority thread pool, and low-priority tasks are processed using a low-priority thread pool. Each thread pool is configured with independent queue capacity, number of threads, and rejection policy.

5. The method according to claim 4, characterized in that, The method further includes: Configure different maximum number of retries and retry intervals for processing tasks with different priorities; Whether to trigger a retry depends on the type of exception, which includes network exceptions, I / O exceptions, and business logic exceptions. When the exception type is a network exception or an I / O exception, a retry is triggered; when the exception type is a business logic exception, a retry is not triggered.

6. The method according to claim 1, characterized in that, The process of converting the processed data into a unified data structure and storing it in a database includes: XML messages are converted into standard data structures using a unified mapping model to adapt to database storage formats; The standard data structure is encapsulated into a message and sent to a message middleware for buffering. Messages are asynchronously retrieved from the message middleware through an independent consumption service, database insertion operations are performed, and reliable data storage is ensured based on a message acknowledgment mechanism.

7. The method according to claim 1, characterized in that, The method further includes: Based on the data stored in the database, generate device status information, operating indicators, and alarm information for visualization. The device status information, operating indicators, and alarm information are provided to the front-end visualization system for display and monitoring in the form of charts.

8. A multi-type XML device data processing apparatus based on Netty, characterized in that, The device includes: The communication access and decoding module is used to receive encoded data packets from the device through a TCP long connection server built with the Netty framework; and to decode and decompress the encoded data packets to obtain data packets in XML format. The data type identification and dynamic routing module is used to parse the XML format data message, extract the type identifier field, and route the XML format data message to the corresponding business processing strategy according to the type identifier field. A multi-level task scheduling module is used to encapsulate the XML format data message into a processing task, and deliver the processing task to the corresponding thread pool for asynchronous processing based on the priority of the business processing strategy. The data abstraction and persistence module is used to transform processed data into a unified data structure and store it in the database.

9. A multi-type XML device data processing device based on Netty, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Including instructions that, when run on a computer, cause the computer to perform the method described in any one of claims 1-7 above.

Images