Data processing method and device based on multi-protocol heterogeneous equipment

Abstract

The invention provides a data processing method and device based on multi-protocol heterogeneous equipment, and relates to the technical field of data processing, and the method comprises the steps: based on a preset connection mode, accessing emergency equipment of at least one target type; acquiring a data stream of each emergency device; the data streams are analyzed, and data of the emergency devices are obtained; and processing the data of each emergency device to obtain a processing result. Through the access of the emergency equipment of at least one target type, the data stream of each emergency equipment can be obtained, then the data stream is analyzed to obtain the data of the emergency equipment, and then the data of the emergency equipment is processed to obtain the processing result, so that the method can be compatible with the emergency equipment of multiple protocol types in an emergency scene, and the reliability of the emergency equipment is improved. Concurrent processing of data of emergency equipment of multiple protocol types is achieved, and the data processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method and apparatus based on multi-protocol heterogeneous devices. Background Technology

[0002] In related technologies, the use of a single protocol access scheme to connect to security emergency equipment, coupled with a customized development model and a synchronous blocking processing architecture, results in poor compatibility for heterogeneous device access. It requires the separate development of interfaces for different protocols, leading to high development costs. Furthermore, the real-time performance of data processing is insufficient, failing to meet the rapid response requirements in emergency scenarios. Concurrency capabilities are limited, making it difficult to support the simultaneous access and management of large-scale emergency equipment.

[0003] Therefore, how to process data from multi-source heterogeneous devices is a key issue. Summary of the Invention

[0004] This invention provides a data processing method and apparatus based on multi-protocol heterogeneous devices to solve the problem of how to process data from multi-source heterogeneous devices.

[0005] This invention provides a data processing method based on multi-protocol heterogeneous devices, comprising: Based on a preset connection method, connect to at least one type of emergency equipment; Obtain the data streams of each of the aforementioned emergency devices; The data streams are parsed to obtain the data of each emergency device. The data from each of the aforementioned emergency devices are processed to obtain the processing results.

[0006] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, wherein the data stream includes a magic number and a port number, and the step of parsing each of the data streams to obtain data from each of the emergency devices includes: For each of the emergency devices' data streams, extract the magic number and the port number from the data stream; Based on the magic number and the port number, the protocol type of the emergency device is identified; The parser corresponding to the protocol type is invoked to parse the data stream and obtain the data of the emergency device.

[0007] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, wherein processing the data of each of the emergency devices to obtain processing results includes: The data of each of the emergency devices is stored in partitions, and the data is entered into a message queue in the form of messages. The data in the message queue is processed to obtain the processing result.

[0008] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, wherein processing the data in the message queue to obtain a processing result includes: Based on the message headers of the messages in the message queue, the data corresponding to the messages in the message queue is classified. The categorized data is processed to obtain the processing result.

[0009] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, wherein classifying the data corresponding to the messages in the message queue based on the message headers of the messages in the message queue includes: Based on the message header of the message in the message queue, determine whether the data corresponding to the message in the message queue is urgent data; Based on the judgment results, the data corresponding to the messages in the message queue are classified.

[0010] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, wherein classifying the data corresponding to messages in the message queue based on the judgment result includes: If the data corresponding to a message in the message queue is urgent data, the data corresponding to the message in the message queue will be placed into the first priority queue. If the data corresponding to a message in the message queue is not urgent, the data corresponding to the message in the message queue is placed into the second priority queue.

[0011] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, the method further comprising: If the number of times urgent data processing in the first priority queue times out or fails exceeds a first preset threshold, an email alert will be triggered. If the number of times non-urgent data processing in the second priority queue times out or fails exceeds the second preset threshold, the non-urgent data in the second priority queue will be recorded to the target queue.

[0012] According to the present invention, a data processing method based on multi-protocol heterogeneous devices is provided, the method further comprising: The processed data is categorized and stored, and data queries are performed based on the processed data.

[0013] According to a data processing method based on multi-protocol heterogeneous devices provided by the present invention, after accessing emergency devices of at least one protocol type, the method further includes: A concurrency control mechanism is used to control the connection links of each of the emergency devices; the concurrency control mechanism is used to switch to the backup service when the connection link of the emergency device is interrupted and the main service is unable to respond due to a fault.

[0014] The present invention also provides a data processing apparatus based on multi-protocol heterogeneous devices, comprising: The access module is used to connect to at least one type of emergency equipment based on a preset connection method. The acquisition module is used to acquire the data streams of each of the emergency devices; The parsing module is used to parse each of the data streams to obtain the data of each of the emergency devices; The processing module is used to process the data from each of the emergency devices to obtain the processing results.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method based on a multi-protocol heterogeneous device as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method based on multi-protocol heterogeneous devices as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method based on multi-protocol heterogeneous devices as described above.

[0018] The present invention provides a data processing method and apparatus based on multi-protocol heterogeneous devices. This method involves connecting to at least one target type of emergency device via a preset connection method; acquiring data streams from each emergency device; parsing each data stream to obtain data from each emergency device; and processing the data from each emergency device to obtain a processing result. By connecting to at least one target type of emergency device, the method can acquire data streams from each emergency device, parse these data streams to obtain the emergency device data, and then process the emergency device data to obtain a processing result. This approach is compatible with emergency devices of multiple protocol types in emergency scenarios, enabling concurrent processing of data from emergency devices of various protocol types and improving data processing efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the data processing method based on multi-protocol heterogeneous devices provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the process for classifying and processing data corresponding to messages in a message queue, provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the architecture of the data processing method based on multi-protocol heterogeneous devices provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the data processing device based on multi-protocol heterogeneous devices provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] To facilitate a clear understanding of the various embodiments of this application, relevant background knowledge will be introduced first.

[0027] I. Related Technologies Using a single protocol access scheme to connect security and emergency equipment, such as a provincial emergency platform that only supports the Message Queuing Telemetry Transport (MQTT) protocol, necessitates the deployment of an additional protocol conversion gateway for field equipment using BeiDou short message service, increasing hardware costs by 30%. Adopting a customized development model, such as a fire data platform that developed five independent access interfaces to adapt to five types of equipment, increases maintenance costs by 25% annually. Using a synchronous blocking processing architecture, such as a smart city platform that uses Tomcat to synchronously process device connections, results in a request timeout rate of 40% when concurrency exceeds 10,000.

[0028] The following drawbacks exist: poor compatibility with heterogeneous devices, requiring separate interface development for different protocols, resulting in high development costs; insufficient real-time data processing, failing to meet the rapid response needs in emergency scenarios; and limited concurrency capabilities, making it difficult to support the simultaneous access and management of large-scale emergency equipment.

[0029] II. Explanation of Relevant Terms Heterogeneous equipment refers to emergency equipment with differences in hardware interfaces, communication protocols, and data formats. It typically includes three types of heterogeneous scenarios: Heterogeneous communication protocols: such as drones supporting Message Queuing Telemetry Transport (MQTT), firefighting robots based on Transmission Control Protocol (TCP), and environmental monitoring sensors using User Datagram Protocol (UDP); Heterogeneous network standards: such as vehicle-mounted terminals with 4G / 5G communication, field equipment with BeiDou short message communication, and sensor nodes with LoRa wireless transmission; Heterogeneous data formats: such as device status data in binary format, alarm information in JSON format, and task instructions in XML format.

[0030] Microservice architecture: This architecture breaks down the cloud platform into independently deployed, loosely coupled service units that work together through standardized application programming interfaces (APIs). Core components include: service registration and discovery (e.g., Consul): dynamically maintaining the network addresses of access service instances and automatically routing requests to backup instances when an instance fails, supporting second-level failover; load balancing (e.g., Nginx): distributing device connection requests based on the least-connection algorithm to avoid single-node overload and improve resource utilization; and containerized deployment (Docker / Kubernetes): encapsulating services into container images and using Kubernetes to achieve automatic scaling of instances (e.g., dynamically adjusting the number of replicas based on the number of device connections).

[0031] Protocol parsing gateway: Middleware developed based on the Netty high-performance network framework. Its core functions include: Protocol adapter: Implementing a parsing and retransmission mechanism for MQTT messages at Quality of Service (QoS) levels 0 / 1 / 2; Implementing custom header parsing (such as a 16-byte magic number + data length field) for TCP / UDP to ensure accurate data boundary identification; Data mapper: Converting data from different protocols into a unified model, such as converting the hexadecimal encoding of BeiDou short messages into geographic coordinate objects.

[0032] Message Queue (Kafka): Employs a distributed log structure and supports a partitioning mechanism: Stores device data in partitions by topic, such as "drone-data" and "sensor-data"; Consumer Groups: Multiple instances consume data from the same topic in parallel, improving processing throughput.

[0033] The following is combined with Figures 1-3 This invention describes a data processing method based on multi-protocol heterogeneous devices.

[0034] Figure 1 This is a flowchart illustrating the data processing method based on multi-protocol heterogeneous devices provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 101-104.

[0035] Step 101: Connect at least one type of emergency device based on a preset connection method.

[0036] It should be noted that the data processing method based on multi-protocol heterogeneous devices provided by the present invention can be applied to the data processing scenario of emergency devices with multiple heterogeneous protocol types in emergency scenarios. The execution subject of the method can be a data processing device based on multi-protocol heterogeneous devices, such as an electronic device, or a control module in the data processing device based on multi-protocol heterogeneous devices for executing the data processing method based on multi-protocol heterogeneous devices.

[0037] Specifically, the preset connection methods include long connections (such as WebSocket long connections) and / or short connections (such as TCP short connections). Long connections are used to transmit real-time video streams from the drone, with a single connection bandwidth of ≥2Mbps; short connections are used to transmit alarm data from sensors, with a connection duration of ≤10 seconds.

[0038] The target type can be a protocol type, network type, or data type. The protocol type can be Message Queuing Telemetry Transport (MQTT), Transmission Control Protocol (TCP), or User Datagram Protocol (UDP). The network type can be 4G, 5G, BeiDou short message communication, or LoRa. Different network types can be switched to each other, and the transmission path can be automatically switched by monitoring the signal strength (e.g., automatically switching to BeiDou short message when the 4G signal is weak).

[0039] The data type can be binary, JSON, or XML format for alarm messages. Emergency equipment can include MQTT-enabled drones, TCP-based firefighting robots, UDP-based environmental monitoring sensors, 4G / 5G communication vehicle terminals, BeiDou short message communication field equipment, and LoRa wireless transmission sensors.

[0040] After the target type of emergency equipment is connected, a TLS 1.3 two-way authentication mechanism can be used to authenticate the emergency equipment. The emergency equipment must provide a pre-configured certificate (including device ID, device type, and validity period). Unauthenticated emergency equipment will be directly rejected to prevent unauthorized access.

[0041] Step 102: Obtain the data stream of each of the aforementioned emergency devices. The data stream may include the magic number and port number. The data stream may also include device status and device location information.

[0042] Specifically, after emergency equipment of various target types is connected, the data stream of each emergency equipment can be obtained in real time.

[0043] Step 103: Parse each of the data streams to obtain the data of each of the emergency devices.

[0044] Specifically, the data from emergency equipment can be equipment status data, alarm information, or task instructions (such as instructions to query historical tasks).

[0045] By parsing each data stream, data from each emergency device can be obtained.

[0046] Step 104: Process the data of each of the emergency devices to obtain the processing results.

[0047] For example, drones (MQTT protocol) transmit disaster area images in real time, which can be parsed and processed before being distributed to the command center's large screen; life detectors (TCP protocol) send detection data every 200ms, which is parsed and processed to mark the location of survivors; Beidou terminals (short message protocol) send coordinate data when communication is interrupted, and the coordinate data can be converted into a standard format through the protocol gateway.

[0048] The data processing method based on multi-protocol heterogeneous devices provided by this invention involves accessing at least one target type of emergency device through a preset connection method; acquiring data streams from each emergency device; parsing each data stream to obtain data from each emergency device; and processing the data from each emergency device to obtain a processing result. By accessing emergency devices of at least one protocol type, the method can acquire data streams from each emergency device, parse the data streams to obtain the data from the emergency devices, and then process the data from the emergency devices to obtain a processing result. This method is compatible with emergency devices of multiple protocol types in emergency scenarios, enabling concurrent processing of data from emergency devices of multiple protocol types and improving data processing efficiency.

[0049] Optionally, after accessing emergency devices of at least one protocol type, the method further includes: A concurrency control mechanism is used to control the connection links of each of the emergency devices; the concurrency control mechanism is used to switch to the backup service when the connection link of the emergency device is interrupted and the main service is unable to respond due to a fault.

[0050] Specifically, after connecting emergency devices of at least one protocol type, a concurrency control mechanism can be used to control the connection links of each emergency device. This involves using the Reactor pattern to handle device connections, with the master reactor (NioEventLoopGroup) responsible for accepting connections and the slave reactors responsible for I / O read / write. Each slave reactor is bound to multiple threads (e.g., 8 threads), capable of handling 100,000 concurrent connections. The access service employs a master-slave deployment. In the event of an interruption in the connection link of an emergency device and the master service becoming unresponsive due to a fault, the service switches to the backup service via Zookeeper.

[0051] Access services (such as the Netty service responsible for TCP / UDP connection management) are deployed in a master-slave configuration to ensure uninterrupted connection between devices and the platform. When the master service becomes unresponsive due to a failure (such as server crash), the master-slave election mechanism of Zookeeper is used to switch to the backup service, ensuring continuous data transmission from the devices to the platform.

[0052] Optionally, the data stream includes a magic number and a port number, and the specific implementation of step 103 above includes: For each of the emergency devices' data streams, the magic number and port number are extracted from the data stream; based on the magic number and port number, the protocol type of the emergency device is identified; the parser corresponding to the protocol type is invoked to parse the data stream and obtain the data of the emergency device.

[0053] Specifically, for each emergency device's data stream, the magic number and port number can be extracted. Based on a preset correspondence and the extracted magic number and port number, the protocol type of the emergency device can be identified. The correspondence represents the relationship between at least one magic number and at least one port number and at least one protocol type. Through dual identification using "magic number + port number" (e.g., BeiDou short message includes magic number 0xAA55 + port 8080, MQTT magic number 0x01 + port 1883), the accuracy of protocol type identification is ensured to reach 100%.

[0054] A parser for each protocol type is pre-configured. After identifying the protocol type of the emergency device, the parser corresponding to that protocol type (such as the BeiDou parser or MQTT parser) can be called to parse the data stream of the emergency device and obtain its data.

[0055] Optionally, after obtaining the data from the emergency equipment, the integrity of the data can be verified by CRC check (e.g., using CRC check for BeiDou short messages) or fixed header (e.g., using a fixed header for MQTT). After the verification is successful, the data is converted into a unified data model (DeviceData).

[0056] Optionally, the specific implementation of step 104 above includes: The data of each emergency device is partitioned and stored, and the data is entered into a message queue in the form of messages; the data in the message queue is processed to obtain the processing result.

[0057] Specifically, the data from each emergency device is partitioned and stored according to a topic, such as "drone-data" and "sensor-data". The data is then entered into a message queue in the form of messages, and the data in the message queue is processed to obtain the processing results.

[0058] Optionally, processing the data in the message queue to obtain a processing result includes: Based on the message headers of the messages in the message queue, the data corresponding to the messages in the message queue is classified. The categorized data is processed to obtain the processing result.

[0059] Specifically, based on the message headers in the message queue, the urgent_flag in the message header is checked, or the message header content is identified. For example, if the message contains the keyword "fire alarm," it is determined to be urgent. The data corresponding to the messages in the message queue is then classified, either as urgent data or ordinary data. The classified data is then processed to obtain the processing results.

[0060] Optionally, classifying the data corresponding to the messages in the message queue based on the message headers of the messages in the message queue includes: Based on the message header of the message in the message queue, determine whether the data corresponding to the message in the message queue is urgent data; based on the determination result, classify the data corresponding to the message in the message queue.

[0061] Specifically, based on the message headers of messages in the message queue, it can be determined whether the data corresponding to the message in the message queue is urgent data, for example, urgent data is classified as urgent data. If the data is determined to be urgent data, the data corresponding to the message in the message queue is classified as urgent data; if the data is determined not to be urgent data, the data corresponding to the message in the message queue is classified as not urgent data. Based on the determination results, the data corresponding to the messages in the message queue can be classified.

[0062] Optionally, classifying the data corresponding to the messages in the message queue based on the judgment result includes: If the data corresponding to a message in the message queue is urgent, the data corresponding to the message in the message queue is placed into the first priority queue; if the data corresponding to a message in the message queue is not urgent, the data corresponding to the message in the message queue is placed into the second priority queue.

[0063] Specifically, the first priority queue can be a high-priority queue, implemented using Disruptor's lock-free queue mechanism to ensure low latency. The second priority queue is a low-priority queue, i.e., a regular queue, implemented using RabbitMQ or Kafka, prioritizing throughput.

[0064] If the data corresponding to a message in the message queue is urgent (such as "equipment battery < 10%" or "personnel calling for help at a fire scene"), then the data corresponding to the message in the message queue is placed in the first priority queue, i.e., the high priority queue. If the data corresponding to a message in the message queue is not urgent (such as "equipment daily inspection status" or "historical trajectory query"), then the data corresponding to the message in the message queue is placed in the second priority queue, i.e., the low priority queue.

[0065] It's important to note that for data in high-priority queues, multiple thread pools can be used (e.g., an 8-thread pool), and a processing timeout threshold can be set, such as 8ms or 10ms. If the processing time of data in the high-priority queue exceeds the timeout threshold, it indicates a timeout, triggering a degradation mechanism that directly returns "processing" and logs a retry. Timeout policies can also be configured, such as automatically triggering a circuit breaker and degradation if data processing is not completed within 10ms (e.g., due to network fluctuations causing computational delays), marking the task as "pending retry," and recording the number of degradation attempts via Sentinel. For data in ordinary queues, multiple thread pools can also be started (e.g., a 4-thread pool), and a processing timeout threshold can be set, such as 100ms. If the processing time of data in the ordinary queue exceeds the timeout threshold, it indicates a timeout, allowing asynchronous retries, but not exceeding the maximum allowed number, such as a maximum of 3 attempts.

[0066] You can also write urgent and regular data to the Redis cache. The key format for urgent data is `data:{device_id}:{timestamp}`, where `timestamp` represents the write time. An expiration time can also be set, for example, 30 minutes (for high-frequency access scenarios). The key format for regular data is `data:{task_id}`, and an expiration time can also be set, for example, 10 minutes (adjustable as needed).

[0067] Next, urgent and non-urgent data are asynchronously written to Elasticsearch (independent of the processing thread) using a separate thread, with batch submissions via the Bulk API (merging every 100 records). Indexes are then created for both urgent and non-urgent data; for example, a hot index is set for urgent data, named `drone_urgent-2025.06` (1 shard + 2 replicas), while a warm index is set for normal data, named `drone_normal-2025.06` (1 shard + 1 replica). Persisting both urgent and non-urgent data to Elasticsearch is a storage strategy designed based on the characteristics of emergency scenario data (real-time performance, retrieval requirements, and correlation). The aim is to achieve efficient data querying, rapid tracing, and multi-dimensional analysis, while balancing the timeliness of urgent data with the long-term value of non-urgent data. Persisting both types of data to Elasticsearch is the optimal solution for balancing "real-time performance, retrieval efficiency, and storage cost" in emergency scenarios, ensuring rapid response to urgent data while meeting the long-term analysis needs of non-urgent data.

[0068] Figure 2 This is a flowchart illustrating the process of classifying and processing data corresponding to messages in a message queue, as provided by the present invention. Figure 2As shown, the method includes steps 201-207.

[0069] Step 201: Based on the message header of the message in the message queue, determine whether the data corresponding to the message in the message queue is urgent data; if the data corresponding to the message in the message queue is urgent data, proceed to step 202; if the data corresponding to the message in the message queue is not urgent data, proceed to step 203.

[0070] Step 202: Place the data corresponding to the message in the message queue into the first priority queue.

[0071] Step 203: Place the data corresponding to the message in the message queue into the second priority queue.

[0072] Step 204: For urgent data in the first priority queue, enable multi-threading for processing. For example, enable an 8-thread pool (CPU cores × 2), set a 10ms timeout threshold (if the timeout occurs, a degradation is triggered, directly returning to "processing" and recording a retry log). If the task is not completed within 10ms (e.g., due to network fluctuations causing computational delays), the circuit breaker is automatically triggered, the task is marked as "pending retry," and the number of degradations is recorded via a Sentine.

[0073] Step 205: For non-urgent data in the second priority queue, enable multi-threading for processing. For example, enable a 4-thread pool (number of CPU cores) and set a 100ms timeout threshold (if timeout occurs, retry asynchronously, up to 3 times).

[0074] Step 206: Write the processing results of urgent data in the first priority queue and non-urgent data in the second priority queue into the Redis cache.

[0075] Step 207: Persist urgent and non-urgent data to Elasticsearch.

[0076] Optionally, the method further includes: If the number of times the urgent data processing in the first priority queue times out or fails exceeds a first preset threshold, an email alarm is triggered; if the number of times the non-urgent data processing in the second priority queue times out or fails exceeds a second preset threshold, the non-urgent data in the second priority queue is recorded to the target queue.

[0077] Specifically, both the first preset threshold and the second preset threshold are pre-set timeout thresholds. For example, the first preset threshold is 2 times and the second preset threshold is 3 times.

[0078] If the number of timeouts or failures in processing urgent data in the first priority queue exceeds a first preset threshold, the timeout task enters the local queue and automatically retryes after 50ms (up to 2 times). If it still fails after more than 2 retries, an email alert is triggered. If the number of timeouts or failures in processing non-urgent data in the second priority queue exceeds a second preset threshold, it is asynchronously retried 3 times. If it still fails after more than 3 retries, the non-urgent data in the second priority queue is recorded to the target queue (e.g., the Dead Letter Queue (DLQ)) for manual inspection and processing.

[0079] You can also use Prometheus to collect metrics such as thread pool queue length, processing time, and Redis hit rate; the success rate of processing urgent data (target value 99.99%) and the average processing time of ordinary data (target <80ms) can be displayed in real time on the Grafana dashboard.

[0080] Optionally, the method further includes: The processed data is categorized and stored, and data queries are performed based on the processed data.

[0081] Specifically, the processed data (such as verified location data) can be associated with structured or unstructured data storage modules through a globally unique identifier (UUID). For example, structured data can be stored in MySQL, unstructured data in MinIO, and time-series data in InfluxDB. Data queries can then be performed based on the processed data, supporting cross-engine queries (such as "the video clip corresponding to a certain emergency alarm").

[0082] Figure 3 This is a schematic diagram of the architecture of the data processing method based on multi-protocol heterogeneous devices provided by the present invention, as shown below. Figure 3 As shown, it includes a device access layer, a protocol parsing layer, a data processing layer, and a service layer.

[0083] The device access layer is used to connect at least one type of emergency equipment based on a preset connection method, providing physical access links for various emergency equipment (drones, robots, Beidou terminals, etc.) and transmitting the raw data (data stream) generated by the equipment to the protocol parsing layer. Emergency equipment establishes a connection with the access layer through its supported communication methods (such as 4G / 5G, Beidou short message service, LoRa, etc.) (TCP connection, UDP message service, WebSocket long connection, etc.), and sends data such as device status and location information to the access layer in the form of a raw data stream (byte stream) (e.g., a hexadecimal byte stream of $BDEPI,1854915,4223618,... sent by a Beidou terminal). The device access layer does not parse the data content; it is only responsible for link maintenance (such as maintaining connection and reconnection mechanisms), authentication (verifying device legitimacy), and data forwarding (transmitting the byte stream completely to the protocol parsing layer), ensuring that the raw data is not lost or tampered with.

[0084] The protocol parsing layer is used to acquire data streams from various emergency devices, extract the magic number and port number from the data streams, identify the protocol type of the emergency devices based on the magic number and port number, and call the parser of the corresponding protocol type to parse the data stream and obtain the data from the emergency devices. Specifically, it reads the data stream and identifies the protocol type (by determining the magic number and port number); then uses the protocol parsing gateway (Netty framework) to call the corresponding parser (such as the MQTT parser for fixed headers + variable headers + payload) to parse the data; subsequently, it performs data verification (such as CRC verification of BeiDou short messages); and finally, it converts the data into a unified data model (DeviceData class).

[0085] The data processing layer partitions and stores data from various emergency devices, then sends the data as messages to a message queue (Kafka). Based on the message headers, it determines whether the data corresponding to a message is urgent. If the data is urgent, it is placed in the first priority queue; otherwise, it is placed in the second priority queue. The urgent data in the first priority queue and the non-urgent data in the second priority queue are processed separately using multi-threading to obtain the processing results. The processing results are written to a Redis cache, and the urgent and non-urgent data are persisted to Elasticsearch. If the number of timeouts or failures in processing urgent data in the first priority queue exceeds a first preset threshold, an email alert is triggered. If the number of timeouts or failures in processing non-urgent data in the second priority queue exceeds a second preset threshold, the non-urgent data in the second priority queue is recorded in the target queue.

[0086] The service layer provides device management services, data storage services, and interface services. The device management service maintains device files (model, protocol type, deployment location) and supports online device status monitoring and remote control command issuance. The data storage service stores processed data by type (structured data to MySQL, unstructured data to MinIO, and time-series data to InfluxDB). The interface service provides data querying (such as "query alarm data for a certain area within 30 minutes") to the command center and third-party systems through RESTful APIs.

[0087] This invention designs a protocol description file (.proto format). The .proto format provides standardized, cross-platform data definitions, facilitating communication between different systems or modules. It also avoids the redundancy and errors of manually writing parsing code. By dynamically loading the protocol parser through Java reflection, it achieves complete decoupling between the parser and the platform's core framework. Reflection allows the program to dynamically inspect classes, create objects, and call methods at runtime (rather than compile time). The core framework dynamically loads and instantiates the parser at runtime based on configuration information (such as class name) using reflection (e.g., `Class.forName()` and `newInstance()`), instead of hard-coding dependencies in the code. The data processing layer employs a four-stage pipeline: protocol parsing, data cleaning, format alignment, and event distribution. Asynchronous thread pools are used for isolation (e.g., 100 threads for the parsing stage, 200 threads for the cleaning stage) to prevent any blockage in one stage from affecting the overall process. Between the protocol parsing layer and the data processing layer, a "priority token bucket" rate-limiting algorithm is introduced: token quotas are allocated based on data type (urgent / normal), and rate-limited data is then distributed to the four-stage pipeline in the data processing layer. 30% of the token quota is allocated to urgent data such as SOS alarms and location data, while 70% is allocated to normal data (e.g., device status inspection). When concurrency exceeds a threshold (800,000 devices / second), rate limiting is only triggered on normal data, and the rate-limited data is temporarily stored in a Kafka persistent queue, automatically processed after the load decreases, ensuring that urgent data is not lost or delayed. The service layer is functionally divided into protocol access service, data conversion service, and device management service, each of which can be scaled up and down independently. Based on Prometheus, the cluster load is monitored, and when the CPU utilization exceeds 70%, a new container instance is automatically created through Kubernetes (average scaling time ≤ 30 seconds).

[0088] This invention supports access via 12 mainstream protocols, and the development cycle for new protocols is shortened from 2 weeks to 2 days; moreover, it is compatible with 85% of emergency equipment on the market, including more than 30 kinds of old emergency equipment (such as GPRS sensors produced before 2010).

[0089] Alternatively, for the above framework, the invention may also provide related alternative modules, as shown in Table 1.

[0090] Table 1. Comparison of Alternative Modules and Their Advantages and Disadvantages

[0091] The data processing apparatus based on multi-protocol heterogeneous devices provided by the present invention is described below. The data processing apparatus based on multi-protocol heterogeneous devices described below and the data processing method based on multi-protocol heterogeneous devices described above can be referred to in correspondence with each other.

[0092] Figure 4 This is a schematic diagram of the data processing device based on multi-protocol heterogeneous devices provided by the present invention, as shown below. Figure 4 As shown, the data processing device 400 based on multi-protocol heterogeneous devices includes: an access module 401, an acquisition module 402, a parsing module 403, and a processing module 404; wherein, Access module 401 is used to connect to at least one type of emergency equipment based on a preset connection method; Acquisition module 402 is used to acquire the data stream of each of the emergency devices; The parsing module 403 is used to parse each of the data streams to obtain the data of each of the emergency devices; The processing module 404 is used to process the data of each of the emergency devices to obtain the processing results.

[0093] The data processing device based on multi-protocol heterogeneous devices provided by this invention connects to at least one target type of emergency device through a preset connection method; acquires data streams from each emergency device; parses each data stream to obtain data from each emergency device; and processes the data from each emergency device to obtain a processing result. By connecting to emergency devices of at least one protocol type, it is possible to acquire data streams from each emergency device, parse the data streams to obtain the data from the emergency devices, and then process the data from the emergency devices to obtain a processing result. This device is compatible with emergency devices of multiple protocol types in emergency scenarios, enabling concurrent processing of data from emergency devices of multiple protocol types and improving data processing efficiency.

[0094] Optionally, the data stream includes a magic number and a port number, and the parsing module 403 is specifically used for: For each of the emergency devices' data streams, extract the magic number and the port number from the data stream; Based on the magic number and the port number, the protocol type of the emergency device is identified; The parser corresponding to the protocol type is invoked to parse the data stream and obtain the data of the emergency device.

[0095] Optionally, the processing module 404 is specifically used for: The data of each of the emergency devices is stored in partitions, and the data is entered into a message queue in the form of messages. The data in the message queue is processed to obtain the processing result.

[0096] Optionally, the processing module 404 is further configured to: Based on the message headers of the messages in the message queue, the data corresponding to the messages in the message queue is classified. The categorized data is processed to obtain the processing result.

[0097] Optionally, the processing module 404 is further configured to: Based on the message header of the message in the message queue, determine whether the data corresponding to the message in the message queue is urgent data; Based on the judgment results, the data corresponding to the messages in the message queue are classified.

[0098] Optionally, the processing module 404 is further configured to: If the data corresponding to a message in the message queue is urgent data, the data corresponding to the message in the message queue will be placed into the first priority queue. If the data corresponding to a message in the message queue is not urgent, the data corresponding to the message in the message queue is placed into the second priority queue.

[0099] Optionally, the data processing device 400 based on multi-protocol heterogeneous devices further includes: The triggering module is used to trigger an email alarm when the number of times the emergency data processing in the first priority queue times out or fails exceeds a first preset threshold. The recording module is used to record non-urgent data in the second priority queue to the target queue when the number of times non-urgent data processing in the second priority queue times out or fails exceeds a second preset threshold.

[0100] Optionally, the data processing device 400 based on multi-protocol heterogeneous devices further includes: The storage module is used to classify and store the processed data, and to perform data queries based on the processed data.

[0101] Optionally, after accessing emergency devices of at least one protocol type, the data processing device 400 based on multi-protocol heterogeneous devices further includes: The control module is used to control the connection links of each of the emergency devices using a concurrent control mechanism; the concurrent control mechanism is used to switch to the backup service when the connection link of the emergency device is interrupted and the main service cannot respond due to a fault.

[0102] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device 500 may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a data processing method based on multi-protocol heterogeneous devices. This method includes: connecting at least one target type of emergency device based on a preset connection method; acquiring data streams from each of the emergency devices; parsing each data stream to obtain data from each of the emergency devices; and processing the data from each of the emergency devices to obtain a processing result.

[0103] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data processing method based on multi-protocol heterogeneous devices provided by the above methods. The method includes: accessing at least one type of emergency device based on a preset connection method; acquiring data streams from each of the emergency devices; parsing each of the data streams to obtain data from each of the emergency devices; and processing the data from each of the emergency devices to obtain a processing result.

[0105] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data processing method based on multi-protocol heterogeneous devices provided by the above methods. The method includes: accessing at least one type of emergency device based on a preset connection method; acquiring data streams from each of the emergency devices; parsing each of the data streams to obtain data from each of the emergency devices; and processing the data from each of the emergency devices to obtain a processing result.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on multi-protocol heterogeneous devices, characterized in that, include: Based on a preset connection method, connect to at least one type of emergency device; Obtain the data streams of each of the aforementioned emergency devices; The data streams are parsed to obtain the data from each of the emergency devices. The data from each of the aforementioned emergency devices are processed to obtain the processing results.

2. The data processing method based on multi-protocol heterogeneous devices according to claim 1, characterized in that, The data stream includes a magic number and a port number. Parsing each data stream to obtain data from each emergency device includes: For each of the emergency devices' data streams, extract the magic number and the port number from the data streams; Based on the magic number and the port number, the protocol type of the emergency device is identified; The parser corresponding to the protocol type is invoked to parse the data stream and obtain the data of the emergency device.

3. The data processing method based on multi-protocol heterogeneous devices according to claim 1, characterized in that, The process of processing the data from each of the emergency devices to obtain the processing result includes: The data of each of the aforementioned emergency devices is stored in partitions, and the data is entered into a message queue in the form of messages. The data in the message queue is processed to obtain the processing result.

4. The data processing method based on multi-protocol heterogeneous devices according to claim 3, characterized in that, The process of processing the data in the message queue to obtain the processing result includes: Based on the message headers of the messages in the message queue, the data corresponding to the messages in the message queue is classified. The categorized data is processed to obtain the processing result.

5. The data processing method based on multi-protocol heterogeneous devices according to claim 4, characterized in that, The step of classifying the data corresponding to the messages in the message queue based on the message headers of the messages in the message queue includes: Based on the message header of the message in the message queue, determine whether the data corresponding to the message in the message queue is urgent data; Based on the judgment results, the data corresponding to the messages in the message queue are classified.

6. The data processing method based on multi-protocol heterogeneous devices according to claim 5, characterized in that, The process of classifying the data corresponding to messages in the message queue based on the judgment result includes: If the data corresponding to a message in the message queue is urgent data, the data corresponding to the message in the message queue will be placed into the first priority queue. If the data corresponding to a message in the message queue is not urgent, the data corresponding to the message in the message queue is placed into the second priority queue.

7. The data processing method based on multi-protocol heterogeneous devices according to claim 6, characterized in that, The method further includes: If the number of times urgent data processing in the first priority queue times out or fails exceeds a first preset threshold, an email alert will be triggered. If the number of times non-urgent data processing in the second priority queue times out or fails exceeds the second preset threshold, the non-urgent data in the second priority queue will be recorded to the target queue.

8. The data processing method based on multi-protocol heterogeneous devices according to claim 1, characterized in that, The method further includes: The processed data is categorized and stored, and data queries are performed based on the processed data.

9. The data processing method based on multi-protocol heterogeneous devices according to claim 1, characterized in that, After accessing emergency devices of at least one protocol type, the method further includes: A concurrency control mechanism is used to control the connection links of each emergency device; the concurrency control mechanism is used to switch to a backup service when the connection link of the emergency device is interrupted and the main service is unable to respond due to a fault.

10. A data processing device based on multi-protocol heterogeneous devices, characterized in that, include: The access module is used to connect to at least one type of emergency equipment based on a preset connection method. The acquisition module is used to acquire the data streams of each of the emergency devices; The parsing module is used to parse each of the data streams to obtain the data of each of the emergency devices; The processing module is used to process the data from each of the emergency devices to obtain the processing results.

Images