Data processing method and system, electronic equipment and computer program product

Abstract

The invention is suitable for the field of data processing, and provides a data processing method and system, electronic equipment and a computer program product, and the method comprises the steps: receiving a plurality of binary data packets transmitted by a plurality of terminal equipment; distributing each binary data packet to a target data analysis channel based on the first metadata of each binary data packet and the load states of a plurality of data analysis channels; controlling the target data analysis channel to analyze the binary data packets according to the target analysis rule of each binary data packet to obtain at least one piece of structured analysis data of each binary data packet; and sending each piece of structured analysis data to one or more target service ends. According to the scheme, the delay of binary data analysis report can be reduced, and the analysis compatibility is improved.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to a data processing method, system, electronic device, and computer program product. Background Technology

[0002] With the rapid development of IoT technology, a massive number of terminal devices (such as smart sensors and smart meters) are being deployed in production and daily life. These terminal devices need to report binary data generated during their operation to the cloud. The increasing number of terminal devices leads to a rise in the format and volume of binary data. The server-side uses a serial mechanism, parsing and sequentially reporting binary data from terminal devices to the business end through a fixed-format parser. This approach suffers from high latency and poor compatibility, making it difficult to meet the real-time monitoring and management needs of terminal devices. Summary of the Invention

[0003] This application provides a data processing method, system, electronic device, and computer program product to solve the problems of high latency and poor compatibility in the prior art due to the increase in data format and data volume of binary data and the reporting of binary data parsing.

[0004] The first aspect of this application provides a data processing method, including: Receive multiple binary data packets transmitted from multiple terminal devices; Based on the first metadata of each binary data packet and the load status of multiple data parsing channels, each binary data packet is assigned to a target data parsing channel; The target data parsing channel is controlled to parse the binary data packet according to the target parsing rule of each binary data packet, so as to obtain at least one structured parsed data of each binary data packet; Each of the structured parsed data is sent to one or more target service terminals.

[0005] A second aspect of this application provides a data processing system, including: The receiving module is used to receive multiple binary data packets transmitted from multiple terminal devices; The allocation module is used to allocate each binary data packet to a target data parsing channel based on the first metadata of each binary data packet and the load status of multiple data parsing channels; The parsing module is used to control the target data parsing channel to parse the binary data packet according to the target parsing rules of each binary data packet, so as to obtain at least one structured parsed data of each binary data packet; The sending module is used to send each of the structured parsed data to one or more target service terminals.

[0006] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0007] A fourth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0008] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] As can be seen from the above, this application receives multiple binary data packets transmitted from multiple terminal devices and allocates these binary data packets based on the first metadata of each binary data packet and the load status of multiple data parsing channels. This achieves dynamic load balancing, improves the parallel processing capability of multiple data packets, and effectively alleviates the high latency problem caused by the increase in data volume. Simultaneously, it controls the target data parsing channel to parse according to the target parsing rules adapted to each binary data packet, improving compatibility with diverse data formats. This allows for the rapid acquisition of structured parsed data, which is then sent to the target service terminal. This solves the problems of high latency and poor compatibility in binary data parsing and reporting caused by increased data formats and volumes in existing technologies. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a server architecture provided in an embodiment of this application; Figure 3 This is a structural diagram of a data processing system provided in an embodiment of this application; Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0017] In specific implementations, the terminals described in the embodiments of this application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0018] The following discussion describes terminals that include displays and touch-sensitive surfaces. However, it should be understood that terminals may include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.

[0019] The terminal supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disc burning applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, exercise support applications, photo management applications, digital camera applications, digital camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0020] Various applications that can run on a terminal can use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the terminal can be adjusted and / or changed between and / or within applications. In this way, the terminal's common physical architecture (e.g., the touch-sensitive surface) can support various applications with user interfaces that are intuitive and transparent to the user.

[0021] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0022] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0023] See Figure 1 , Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application. For example... Figure 1 As shown, a data processing method includes the following steps: Step 101: Receive multiple binary data packets transmitted from multiple terminal devices.

[0024] Terminal devices refer to devices capable of data acquisition and binary data reporting, such as sensors, industrial controllers, and IoT terminals. After acquiring data such as temperature, humidity, and equipment operating status, terminal devices convert this data into binary data and report it as binary data.

[0025] To achieve centralized parallel processing of binary data and improve data parsing and reporting efficiency, the server and terminal devices pre-agree on rules, instructing the terminal devices to package multiple binary data into binary data packets within a short time window, and then report multiple binary data in a centralized manner in the form of binary data packets.

[0026] A binary data packet is a data carrier encapsulated by a terminal device according to a preset protocol. It contains primary metadata describing its overall attributes and multiple binary data entries carrying business information.

[0027] The first metadata includes device identifier, packet generation timestamp, total packet length, start and end offsets of each binary data segment, binary data format, checksum (such as Cyclic Redundancy Check (CRC) value), protocol version number, and other data description information, which are used to support packet identification, parsing, and verification.

[0028] The server receives binary data packets transmitted from various terminal devices and performs binary data packet parsing and data reporting.

[0029] In some embodiments, the server is deployed with a network access layer that supports Message Queuing Telemetry Transport (MQTT) or Hypertext Transfer Protocol (HTTP) to ensure efficient access of a large number of binary data packets in a short period of time and achieve high throughput reception.

[0030] In some embodiments, upon receiving a binary data packet, the binary data packet is stored in a data packet buffer. The data packet buffer is a shared circular memory buffer or memory pool to avoid disk I / O latency and improve reception throughput. The data packet buffer manages data according to the first-in, first-out (FIFO) principle, ensuring that the earliest received binary data packet is allocated for parsing first.

[0031] Packet caching reduces latency through peak shaving and valley filling mechanisms: when a terminal device transmits a large number of binary data packets (peaks) in a short period of time, the packet caching stores these data packets in advance to avoid processing bottlenecks in the subsequent parsing channel due to sudden load increases; when the number of binary data packets transmitted by the terminal device decreases (valleys), the packet caching releases the stored binary data packets in an orderly manner to the subsequent execution process, ensuring that the data parsing channel continues to operate efficiently.

[0032] By decoupling the receiving and distribution of data packets through packet buffering, receiver blocking caused by scheduling delays is avoided, thereby improving the ability to withstand fluctuations.

[0033] By supporting a high-throughput access layer and packet caching for MQTT / HTTP, it meets the high-concurrency receiving requirements of terminal devices in batch reporting scenarios, avoiding data loss or access delay.

[0034] In some embodiments, before storing binary data packets in the data packet buffer, a lightweight check is performed on the binary data packets to verify whether they meet the most basic format and integrity requirements. For example, the header format is checked to ensure it conforms to a preset protocol (e.g., whether the header identifier is correct); the packet length is checked to ensure it matches the total length of the data packets recorded in the first metadata to avoid data truncation; and the CRC value or simple signature is checked to ensure it is valid and that the data packets have not been tampered with.

[0035] The verified binary data packets are stored in the data packet cache to achieve pre-filtering. This ensures that the cache space is used to serve data packets that meet the verification requirements, preventing data packets that do not meet the verification requirements from occupying cache resources and ensuring that cache resources are concentrated on serving verified binary data packets.

[0036] In some embodiments, after storing the binary data packet in the data packet buffer, a lightweight check is immediately performed on the binary data packet to verify whether it meets the most basic format and integrity requirements.

[0037] Retain valid binary data packets in the data packet cache and delete invalid binary data packets, thus implementing post-filtering. Prioritize data reception efficiency and avoid data access delays caused by verification operations. At the same time, quickly filter out invalid or abnormal data to avoid occupying the parsing resources of subsequent data parsing channels.

[0038] Lightweight validation quickly filters out invalid, abnormal, or tampered data packets, such as those with incorrect formats or CRC mismatches, preventing such packets from consuming parsing resources in subsequent data parsing channels, while ensuring operational stability and efficiency under high concurrency input.

[0039] In this application, binary data packets are parsed through a data parsing channel. The data parsing channel is a processing unit with independent thread resources used for parsing binary data packets.

[0040] In some embodiments, after receiving multiple binary data packets transmitted by multiple terminal devices, the method further includes: storing the multiple binary data packets in a data packet buffer; monitoring in real time the number of accumulated binary data packets in the data packet buffer and the channel utilization of the multiple data parsing channels; if the number of accumulated data packets exceeds the upper limit of the number of accumulated data packets and the channel utilization is continuously higher than the high load threshold, then increasing the number of data parsing channels; if the number of accumulated data packets is lower than the lower limit of the number of accumulated data packets and the channel utilization is continuously lower than the low load threshold, then decreasing the number of data parsing channels.

[0041] The backlog refers to the number of binary data packets in the packet buffer that have not been allocated to the data parsing channel.

[0042] Channel utilization rate refers to the proportion of resources used by all data parsing channels.

[0043] In some embodiments, channel utilization = number of packets currently being processed by all channels / total number of packets that can be processed by all channels × 100%.

[0044] The upper and lower limits of the stacking quantity are preset stacking quantity thresholds.

[0045] The high load threshold and low load threshold are preset channel utilization thresholds. The high load threshold refers to the upper limit of the data parsing channel utilization when it is under high load (e.g., 80%), and the low load threshold refers to the lower limit of the data parsing channel utilization when it is under low load (e.g., 30%).

[0046] In some embodiments, the number of binary data packets accumulated in the data packet buffer and the channel utilization of the data parsing channel are monitored in real time.

[0047] In some embodiments, if the backlog exceeds the upper limit and the channel utilization rate remains above the high load threshold (e.g., the high load threshold is 80%, and the channel utilization rate remains at 90% for 5 seconds), channel expansion is triggered. The expansion method is horizontal expansion, i.e., increasing the number of data parsing channels, such as increasing the number of data parsing channels from 5 to 8. The newly added data parsing channels can immediately participate in the allocation and parsing of binary data packets.

[0048] In some embodiments, if the backlog is less than the lower limit of the backlog and the channel utilization remains below the low load threshold (e.g., the low load threshold is 30%, and the channel utilization remains below 20% for 5 seconds), channel scaling down is triggered. The scaling down method is vertical shrinkage, which involves migrating unfinished binary data packets from data parsing channels with low channel utilization (e.g., data parsing channels with channel utilization < 20%) to other data parsing channels, and then closing the data parsing channel, such as reducing from 8 channels to 5, to release thread and memory resources.

[0049] In some embodiments, data parsing latency, i.e. the total time from when a binary data packet enters the data parsing channel to when parsing is completed, can also be monitored. Based on the data parsing latency, the number of packets piling up, and the channel utilization, a comprehensive judgment is made to achieve adaptive scaling management of the data parsing channel and ensure a balance between processing efficiency and resource utilization.

[0050] The duration threshold (e.g., 5 seconds) for expansion and contraction logic can avoid frequent scaling due to instantaneous data fluctuations and improve the stability of data packet allocation.

[0051] The expansion and contraction are designed differently: horizontal expansion focuses on increasing processing capacity by increasing the number of channels, while vertical contraction focuses on optimizing inefficient channels by integrating resources. The two methods take effect independently for high-load and low-load scenarios, avoiding resource allocation chaos or task scheduling conflicts caused by overlapping operation logic.

[0052] It should be noted that expansion and contraction adopt a mutually exclusive triggering mechanism: only one operation is executed at a time, that is, expansion does not trigger contraction, and contraction does not interrupt expansion, so as to avoid conflicts between the number of channels and resource allocation and ensure the stability of the data parsing process.

[0053] In some embodiments, the server uses a dispatcher to adaptively scale the data parsing channel.

[0054] By real-time monitoring and adaptive scaling, data parsing channels are increased during peak data periods to handle backlogs (avoiding parsing latency), and data parsing channels are reduced during off-peak data periods to free up resources (avoiding resource waste), thus achieving on-demand resource allocation and balancing throughput and resource utilization.

[0055] Step 102: Based on the first metadata of each binary data packet and the load status of multiple data parsing channels, each binary data packet is assigned to a target data parsing channel.

[0056] Load status refers to the current resource usage of the data parsing channel. Quantitative indicators include the number of data packets currently being processed by the channel, thread utilization, and cache queue depth, which are used to determine whether the data parsing channel has the ability to handle new binary data packets.

[0057] In some embodiments, a scheduler is used to perform the allocation of binary data packets.

[0058] In some embodiments, the first metadata of binary data packets in the data packet buffer is read in batches to determine data description information such as device identifier, data type, and / or data priority of each binary data packet to be allocated. Simultaneously, the load status of multiple data parsing channels is obtained.

[0059] Based on the device identifier, data type, and / or data priority information in the first metadata, as well as the load status of multiple data parsing channels, a target data parsing channel is matched for each binary data packet, and each binary data packet is assigned to the target data parsing channel. The target data parsing channel is a data parsing channel selected from multiple data parsing channels that is suitable for parsing the current batch of binary data packets.

[0060] By combining primary metadata with channel load status, high-priority / complex binary data packets are avoided from being assigned to overloaded channels, improving parsing efficiency, reducing data packet accumulation on individual channels, balancing channel resource usage, making the number of data packets more compatible with the channel's processing capacity, and reducing parsing latency.

[0061] In some embodiments, allocating each binary data packet to a target data parsing channel based on the first metadata of each binary data packet and the load status of multiple data parsing channels includes: dividing the multiple binary data packets into at least one group to be allocated according to the device identifier, data type, and / or data priority in the first metadata of each binary data packet; determining, from the multiple data parsing channels, a target data parsing channel whose load status meets the load requirements of the group to be allocated for each group to be allocated; and allocating the binary data packets in the group to be allocated to the matching target data parsing channel.

[0062] The device identifier is information in the first metadata used to uniquely identify the terminal device.

[0063] The data type is the business data category carried by the binary data packet (such as temperature data, pressure data, fault code data).

[0064] Data priority refers to the priority level set for data packets based on business importance (e.g., emergency fault data has high priority, and normal status data has low priority), which is used to prioritize the allocation of high-priority binary data packets.

[0065] A pending group refers to a set of data packets divided according to device identifier, data type, and / or data priority. Binary data packets in the same pending group have the same or similar parsing requirements.

[0066] Read the first metadata of each binary data packet to determine its device identifier, data type and / or data priority, etc., in order to divide it into groups to be assigned.

[0067] In some embodiments, the partitioning rules include, but are not limited to, the following: partitioning by device identifier, grouping binary data packets from the same device into one group; partitioning by data type, grouping binary data packets of the same data type into one group; partitioning by device identifier + data type, grouping binary data packets of the same data type from the same device into one group; partitioning by data priority, ensuring that high-priority binary data packets are parsed as early as possible, such as high-priority binary data packets in their own group, and multiple low-priority binary data packets grouped into one group; and so on.

[0068] Using data description information, multiple binary data packets are divided into one or more groups to be assigned, and each group to be assigned contains at least one binary data packet.

[0069] In some embodiments, binary data packets within each group have the same parsing rule requirements. For example, binary data packets of the same data type that correspond to the same parsing rule requirements are grouped together to ensure that the same parsing rules can be reused for subsequent parsing, avoiding frequent switching of parsing rules, improving parsing efficiency, and reducing parsing latency.

[0070] In some embodiments, the load requirement for each group to be assigned is calculated (e.g., number of packets in the group × time taken to parse a single packet = load requirement).

[0071] In some embodiments, for each group to be assigned, all data parsing channels are traversed, and candidate channels whose load status meets the load requirements of the group to be assigned are selected. Then, a target data parsing channel is determined for each group to be assigned from the candidate channels.

[0072] If there are multiple candidate channels for the group to be allocated, a load balancing strategy can be used to select one. For example, if the utilization rate of data parsing channel A is 40% and the utilization rate of data parsing channel B is 50%, data parsing channel A can be used as the target data parsing channel for the group to be allocated.

[0073] If there are no candidate channels in the group to be assigned, the group to be assigned can be split into new groups based on the load status of the data parsing channel to achieve matching and assignment; or, the matching and assignment can be performed after the data parsing channel becomes idle; or, the matching and assignment can be performed after adding a data parsing channel.

[0074] All binary data packets in the group to be assigned are allocated to the target data parsing channel.

[0075] By dividing data into groups to be allocated, scattered binary data packets are aggregated into sets with clear requirements, enabling centralized allocation of data packets with the same attributes. This transforms single-packet allocation into group allocation, reducing the number of allocations and improving scheduling efficiency. Precise matching between load demand and load status ensures that the groups to be allocated can be fully handled by the channels, avoiding parsing interruptions or delays due to insufficient channel capacity. This achieves efficient utilization of parsing resources, prevents overload of a single data parsing channel, and prevents other data parsing channels from becoming idle, thus achieving multi-channel load balancing.

[0076] Parallel parsing is achieved by utilizing multiple data parsing channels, thereby improving the overall parsing throughput and reducing parsing latency.

[0077] Step 103: Control the target data parsing channel to parse the binary data packet according to the target parsing rule of each binary data packet, and obtain at least one structured parsed data of each binary data packet.

[0078] The target parsing rules are translation rules used to parse binary data packets. They define the mapping relationship between binary byte streams and structured fields, such as a byte segment corresponding to "temperature value" and a bit field corresponding to "device status". They support complex parsing operations such as zero-copy field mapping, bit field parsing, and nested table parsing.

[0079] Structured parsed data refers to data that is parsed and can be understood by the business side. It includes target business fields (core data required by the business) and third-party metadata (metadata describing the parsing process and data attributes).

[0080] The target data parsing channel reads the first metadata of the allocated binary data packet and determines the rule identifier of at least one target parsing rule used to parse multiple binary data in it, so as to quickly obtain the target parsing rule based on the rule identifier.

[0081] In some embodiments, the target parsing channel is a parsing rule indicated by a rule identifier obtained from a local rule base or a remote rule base, based on a rule identifier.

[0082] The local rule base supports hot reloading, canary releases, rollback, and version coexistence. Hot reloading: Updates to the local rule base do not require a restart; the updated parsing rules take effect immediately without interrupting data parsing. Canary releases: New parsing rules are first tested on a subset of channels / data, and then fully rolled over after verification. Rollback: If an anomaly occurs with a new parsing rule, it quickly reverts to the previous stable version to avoid parsing failures. Version coexistence: New and old parsing rules temporarily coexist, achieving a smooth transition.

[0083] In some embodiments, the local rule base may adopt a Least Recently Used (LRU) eviction policy, which automatically deletes parsing rules that have not been used for a long time, ensuring that the local rule base caches only frequently used parsing rules.

[0084] The remote rule base is a centralized storage and management center for parsing rules. Its rule sources include: device binary protocol parsing rules defined by R&D personnel (using configurable descriptions, such as JavaScript Object Notation (JSON) / YAML (YAML Ain't Markup Language) format); and new protocol rules updated or issued by the manufacturer, which are registered to the remote rule base through the Application Programming Interface (API).

[0085] The remote rule base can be deployed on a database or configuration center, such as a distributed key-value store (etcd, suitable for high-consistency configuration storage), a service discovery and configuration tool (Consul, which combines key-value storage and configuration distribution capabilities), or a remote dictionary service (Redis, which supports high-performance key-value configuration caching), to achieve unified management and version control of parsing rules. The remote rule base supports parsing rules defined in JSON / YAML format, as well as parsing rules pre-compiled into Web Assembly Language (WebAssembly, WASM) / Low Level Virtual Machine (LLVM) formats.

[0086] In some embodiments, if a parsing rule in the remote rule base is updated (e.g., upgraded), the remote rule base sends an update notification to the server via a proactive push mechanism. Upon receiving the update notification, the server retrieves the latest parsing rule and overwrites the old version in the local rule base, thereby ensuring real-time synchronization between the local and remote rule bases and effectively avoiding parsing errors caused by inconsistent rule versions.

[0087] In some embodiments, the parsing rules in both the local rule base and the remote rule base are version controlled and signed to ensure security.

[0088] In some embodiments, if the target parsing rule is cached in the local rule base, it is invoked directly.

[0089] In some embodiments, when the target parsing rule comes from a remote rule base, the target parsing rule is stored in the local rule base and then invoked. Loading the rule locally via a hot-loading mechanism eliminates the need to restart the parsing service, and the new parsing rule takes effect immediately, reducing parsing latency. Furthermore, caching a copy of the parsing rule locally reduces the overhead of frequent accesses to the remote rule base.

[0090] By using dynamic rule matching and hot reloading, the system enables flexible updates and version management of parsing rules, allowing for adaptation to new device protocols or rule optimizations without requiring a restart, thus improving scalability.

[0091] After the target data parsing channel obtains the target parsing rule corresponding to the binary data packet, it parses the binary data packet according to the target parsing rule and obtains the structured parsing data after the format conversion of the binary data packet.

[0092] In this application, target parsing rules are matched for each binary data packet to complete the structured conversion of the binary data packet, meet the parsing requirements of different data formats of different devices, have multi-data format compatibility, and ensure that data of various data formats can be parsed stably.

[0093] In some embodiments, each binary data packet contains at least one binary data, each binary data contains second metadata and a business field, and controlling the target data parsing channel to parse the binary data packet according to the target parsing rules of each binary data packet to obtain at least one structured parsed data of each binary data packet includes: extracting the business field and the second metadata corresponding to the business field from the binary data packet based on the target parsing rules of the binary data packet by the target data parsing channel; converting the business field into a target business field of a set format, and generating third metadata based on the target business field and the second metadata to obtain the structured parsed data containing the target business field and the third metadata; the third metadata includes data related to the target business field in the second metadata and descriptive data added when generating the target business field.

[0094] Secondary metadata refers to the metadata in binary data used to describe business fields. For example, if the business field is "temperature value 25.4℃", the corresponding secondary metadata includes the data generation timestamp (2024-05-20 09:59:59), data length (2 bytes), source sensor number (S001), data precision (0.1℃), etc.

[0095] Business fields refer to the raw business data carried in binary data that has not been formatted and is in binary format (such as the bit field 0b101 corresponding to the fault status). They need to be converted into target business fields that can be read by the business through the target parsing rules. The target business field refers to the field obtained after format conversion of the business field, which meets the requirements of the target business end.

[0096] The newly added descriptive data refers to the descriptive data added during the generation of the target business fields, describing the parsing process and the target business fields. It differs from the secondary metadata derived from binary data. The newly added descriptive data includes, for example, a conversion completion timestamp, the target data parsing channel number, the target parsing rule version number, and data format conversion identifiers (such as "binary → decimal"). This newly added descriptive data can be used for subsequent data tracking, error retries, and auditing operations.

[0097] After the target data parsing channel loads the target parsing rules, it first splits multiple binary data packets from the binary data packet according to the start and end offsets defined by the first metadata.

[0098] For each piece of binary data, extract its business fields and secondary metadata according to the target parsing rules, and convert the business fields into target business fields.

[0099] Data related to the target business field (such as sensor number, raw data generation time, data type, etc.) is filtered from the second metadata. At the same time, during the business field conversion process, descriptive data corresponding to the target business field will be generated, such as parsing time, parsing completion timestamp (e.g., 2024-05-20 10:00:01), target data parsing channel number for processing the data (e.g., W002), version number of the target parsing rule (e.g., V1.2), data format conversion type (e.g., 16-bit unsigned integer → decimal with unit), global tracing representation (Trace Identifier), etc.

[0100] The descriptive data selected from the second metadata and the newly added descriptive data are combined to obtain the third metadata. The third metadata retains the key attributes of the original data and supplements the key information of the parsing process (the newly added descriptive data), providing a basis for subsequent data auditing, retries and fault location.

[0101] The target business fields and third-party metadata are assembled into structured parsed data according to the preset structure in the target parsing rules. The structured parsed data is the standardized structure defined by the schema registry. This structure can be serialized, stored, and transmitted using structured format containers such as JSON key-value pairs (e.g., { "deviceId": "123", "temp": 25.4, "status": "ok"}), Avro, or Protocol Buffers (Protobuf).

[0102] Structured parsing decouples data from the specific device's protocol and data format, while maintaining clear field semantics and scalability, facilitating unified consumption by subsequent business units.

[0103] This application employs a necessary field validation mechanism to ensure that the structured parsed data contains the minimum information set required for business consumption. Specifically, this includes: pre-marking necessary business fields in the target parsing rules; automatically checking whether the pre-marked necessary fields are missing in the business field after the target data parsing channel extracts the business field; if a necessary field is detected to be missing, the binary data is marked as abnormal data and transferred to the Dead-Letter Queue (DLQ) to prevent invalid data from flowing into subsequent business terminals.

[0104] Step 104: Send each of the structured parsed data to one or more target service terminals.

[0105] The target business end refers to the business end that consumes structured parsed data, such as the monitoring system, device management system, and data analysis system of the Internet of Things platform. Different target business ends may require different structured parsed data.

[0106] The parsed structured data is then sent to one or more target service terminals. This provides standardized, directly usable data input for each service terminal, quickly supporting subsequent business processing.

[0107] In some embodiments, sending each of the structured parsed data to one or more target service terminals includes: performing data formatting processing on each of the structured parsed data to obtain data to be sent; merging multiple data to be sent into at least one data packet to be sent according to the data type and / or service terminal identifier of each data packet to be sent, and configuring a routing identifier of the target service terminal for each data packet to be sent; and sending the data packets to be sent in parallel to the corresponding target service terminals according to the routing identifier.

[0108] Data formatting refers to preprocessing operations performed on structured parsed data, including data merging, field optimization, data encryption, and compression. Data merging: Combining multiple data points collected by the same device at different times into a single message batch for transmission reduces the number of transmissions and network load. Field optimization: Implementing field filtering and optimization to remove redundant data and retain necessary fields, thereby reducing data packet size. For example, infrequently changing fields (such as device model, manufacturer information, etc.) may be removed. Data encryption and compression: Encrypting and compressing data ensures secure and efficient transmission while reducing network bandwidth usage.

[0109] The data to be sent refers to the structured parsed data after data formatting processing.

[0110] Business-side identifiers refer to the identification information used to uniquely identify the target business side. They are third-party metadata derived from structured parsed data and are used to distinguish different delivery objects.

[0111] The data packets to be sent refer to the collection of data packets merged according to data type and / or business terminal identifier. It contains multiple data packets to be sent. The purpose of merging is to reduce the number of deliveries in a single delivery and reduce network overhead.

[0112] Routing identifiers are network address information used to locate the target service, ensuring that data packets to be sent can be accurately delivered to the target service.

[0113] Each structured parsed data is formatted to obtain the data to be sent, thereby improving data delivery efficiency and security.

[0114] In some embodiments, data to be sent is stored in a pre-planned memory buffer, awaiting entry into a lightweight message queue or topic partition (such as a Kafka-style partition). The memory buffer is used to store scattered data to be sent. When the amount of data is large enough, this data is written to the queue or partition in batches, avoiding frequent interactions of "writing one and waiting one time", improving write efficiency and reducing overhead.

[0115] In some embodiments, the data to be sent is written to a lightweight message queue or topic partition. The queue / partition is divided into topics according to the business identifier or data type. For example, data from the monitoring system is written to "monitor_topic", and data from the device management system is written to "device_mgmt_topic".

[0116] In some embodiments, the queue / partition employs an asynchronous write mechanism to decouple parsing and reporting. After writing data, there is no need to wait for the delivery result before processing the next batch of data, avoiding parsing blocking. Data to be sent in the queue / partition is retrieved according to principles such as first-in-first-out or priority scheduling.

[0117] In this application, parsing and reporting are decoupled, which facilitates operations such as batch merging, parallel sending, and traffic suppression, and improves the fault tolerance of each processing stage.

[0118] In some embodiments, data to be sent can be merged based on data type and / or service terminal identifier. Specifically, data is first grouped by service terminal identifier, such as grouping all data to be sent to the monitoring system as one group; within the same group of service terminal identifiers, it is further subdivided by data type, such as dividing it into temperature data and status data, and data to be sent of the same data type is merged into one data packet to be sent.

[0119] In some embodiments, a data packet size threshold (e.g., 1MB) is set during merging. If the amount of data to be sent of a certain data type exceeds the threshold, it is split into multiple data packets to be sent. For example, a 1.5MB data packet to be sent is split into a 1MB data packet to be sent and a 0.5MB data packet to be sent, so as to avoid transmission timeouts caused by excessively large data packets.

[0120] In some embodiments, a routing configuration table is pre-set, in which service identifiers are mapped one-to-one with routing identifiers. The routing configuration table is used to find the routing identifier corresponding to the service identifier of the data packet to be sent, and the routing identifier is configured for the data packet to be sent. Then, the data packet to be sent can be sent to the target service based on the routing identifier.

[0121] In some embodiments, multiple data packets to be sent are sent in parallel using a multi-threaded or multi-channel approach to avoid overall sending delays caused by single-service-end blocking, thereby improving delivery throughput and achieving high concurrency and low latency.

[0122] By accurately locating data packets using routing identifiers, we can prevent data packets from being delivered to the wrong service end and ensure the accuracy of delivery.

[0123] In some embodiments, a scheduler performs batch aggregation on the data to be sent in the queue / partition, that is, merges multiple data packets to be sent from the same target service into a single data packet to be sent, and configures a routing identifier for each data packet. Subsequently, the scheduler employs a parallel delivery mechanism, starting multiple delivery threads / channels to send the data. Each thread / channel is responsible for delivering the data packets to be sent from one target service. For example, thread 1 delivers the data packets to the monitoring system, thread 2 delivers them to the device management system, and thread 3 delivers them to the data analysis system. Each thread / channel runs independently and does not block the others.

[0124] In some embodiments, a retry strategy is enabled during delivery: if a delivery fails (e.g., the target service does not respond within a timeout period), the delivery is retried according to the exponential backoff principle (e.g., the first retry interval is 1 second, the second retry interval is 2 seconds, the third retry interval is 4 seconds, and the maximum number of retries is 5). Data packets that fail to be sent during retry are placed in a dead-letter queue, awaiting manual or automatic replay processing. This retry strategy effectively addresses instantaneous network fluctuations or temporary service failures, ensuring the reliability and real-time performance of large-scale device data reporting.

[0125] In some embodiments, data that fails to be parsed or delivered will be transferred to a dead-letter queue. The dead-letter queue synchronously records the complete original data packet and parsing context information (such as parsing rule version, processing channel identifier, failure node, etc.) of the data, so as to facilitate manual intervention or automatically triggered replay processing, offline re-parsing and other operations on the failed data. At the same time, a failure alarm mechanism can be configured to respond quickly to anomalies.

[0126] This application provides a complete basis for tracing the entire data processing chain, locating faults, and auditing changes in parsing rules by retaining full-chain trace logs and audit records of changes in parsing rules, thereby further ensuring the maintainability of the system and the ability to trace back problems.

[0127] In some embodiments, an idempotency identifier is generated for each data packet to be sent based on the device identifier, timestamp, and sequence number; the idempotency identifier is bound to the data packet to be sent; the data packet to be sent with the idempotency identifier bound is sent to the target service terminal, so that the target service terminal checks the service processing status of the data packet to be sent based on the idempotency identifier, and performs service processing on the data packet to be sent if the service processing status is unprocessed.

[0128] An idempotent identifier is a unique identifier used to prevent duplicate data packets from being sent. It is unique and immutable (the idempotent identifier of the same data packet is always the same).

[0129] In some embodiments, the idempotent identifier is generated by a hash value of a combination of device identifier, timestamp, and serial number.

[0130] A sequence number is a unique, ordered identifier assigned to a data packet to be sent.

[0131] In some embodiments, sequence numbers are generated incrementally along the dimension of device identifier + timestamp. For example, the sequence number of data packets to be sent by the same device within the same minute is incremented from 1 to ensure that data packets to be sent by the same device within the same timestamp are not duplicated.

[0132] The business processing status refers to the processing status of the data packets to be sent, including unprocessed, processing, and processing completed status, which is used to determine whether the data packets to be sent need to be processed.

[0133] After merging the data packets to be sent, extract the core information from the data packets to be sent, namely the device identifier, timestamp, and serial number.

[0134] In some embodiments, a hash algorithm is used to hash the combined string of device identifier, timestamp, and sequence number into a string, which serves as the idempotent identifier for the data packet to be sent.

[0135] In some embodiments, an idempotency flag is written into the header of the data packet to be sent, ensuring that the idempotency flag is tightly bound to the data packet and transmitted together with the data packet body, providing a core basis for the receiving end to perform idempotency verification through the flag and avoid duplicate processing.

[0136] After the target business receives the data packet to be sent bound with the idempotency identifier, it first queries the idempotency processing record table on the server (which stores the idempotency identifier and its corresponding business processing status).

[0137] If the business processing status of the idempotent flag is unprocessed, the business processing will be executed normally (such as writing to the database or triggering monitoring alarms). After the processing is completed, the business processing status of the idempotent flag will be changed to the processing completed status.

[0138] If the idempotent flag indicates that the service processing status is already processed, then the data packet to be sent is discarded and not processed again.

[0139] If the idempotent flag's service processing status is "processing," then wait a preset time (e.g., 5 seconds) before querying again. If the status changes to "processed" after the second query, the data packet to be sent is discarded; otherwise, the service processing is executed normally, and the idempotent flag's service processing status is changed to "processing completed" upon completion.

[0140] By using idempotent flags, the problem of duplicate processing of pending data packets caused by network jitter and retry delivery is fundamentally solved. If the same pending data packet is retried multiple times due to timeout, the target business end only processes it once, avoiding business anomalies such as duplicate database writes and duplicate alarms. Through the idempotent processing record table, the target business end has the ability to trace the business processing status, ensuring the consistency of data processing, such as not processing duplicate data packets when receiving the same data packet concurrently.

[0141] Figure 2 This is a schematic diagram of a server architecture provided in an embodiment of this application. For example... Figure 2 As shown, the server includes processing units such as packet caching, scheduler, data parsing channel, memory buffer, queue / partition, and local rule base to support parallel data parsing and parallel reporting, reduce latency, and improve parsing compatibility.

[0142] In this application, by decoupling and adapting parsing rules to device protocols, flexible expansion of device types is supported, avoiding the need to modify or rewrite the parser when adding new devices, significantly simplifying the deployment process and reducing maintenance costs. The decoupling design of parsing and business reporting breaks the rigid binding of traditional data processing chains, improving business adaptation flexibility and making the data processing flow more scalable and adjustable. Simultaneously, by establishing a unified compression processing, concurrent parsing, and asynchronous reporting mechanism for batch binary data, resource allocation strategies are optimized, effectively improving resource utilization and the overall throughput and efficiency of data processing and reporting.

[0143] This application achieves high-throughput reception, load balancing, flexible parsing, and secure and efficient delivery of binary data by optimizing the entire process of receiving, allocating, parsing, and sending, as well as auxiliary mechanisms such as adaptive scaling, idempotent processing, and fault-tolerant replay. It comprehensively solves the technical problems of low concurrency, poor scalability, easy duplication, and weak fault tolerance in traditional processing methods, and can be widely applied to large-scale binary data processing scenarios such as the Internet of Things and the Industrial Internet.

[0144] In this embodiment, multiple binary data packets transmitted from multiple terminal devices are received, and the binary data packets are allocated based on the first metadata of each binary data packet and the load status of multiple data parsing channels to achieve dynamic load balancing, improve the parallel processing capability of multiple data packets, and effectively alleviate the high latency problem caused by the increase in data volume. Simultaneously, the target data parsing channel is controlled to parse according to the target parsing rules adapted to each binary data packet, improving compatibility with diverse data formats. Structured parsed data can be quickly obtained and sent to the target service terminal, solving the problems of high latency and poor compatibility in binary data parsing and reporting caused by increased data format and volume in existing technologies.

[0145] See Figure 3 , Figure 3 This is a structural diagram of a data processing system provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0146] The data processing system 300 includes: a receiving module 301, an allocation module 302, a parsing module 303, and a sending module 304.

[0147] The receiving module 301 is used to receive multiple binary data packets transmitted by multiple terminal devices.

[0148] The allocation module 302 is used to allocate each binary data packet to a target data parsing channel based on the first metadata of each binary data packet and the load status of multiple data parsing channels.

[0149] The parsing module 303 is used to control the target data parsing channel to parse the binary data packet according to the target parsing rules of each binary data packet, so as to obtain at least one structured parsed data of each binary data packet.

[0150] The sending module 304 is used to send each of the structured parsed data to one or more target service terminals.

[0151] In some embodiments, the allocation module is specifically used for: Based on the device identifier, data type, and / or data priority in the first metadata of each binary data packet, the plurality of binary data packets are divided into at least one group to be assigned; From the plurality of data parsing channels, determine the target data parsing channel whose load status satisfies the load requirements of the group to be allocated for each of the multiple data parsing channels; The binary data packets in the group to be assigned are assigned to the matching target data parsing channel.

[0152] In some embodiments, the system further includes a cache monitoring module, used for: Store multiple binary data packets in a data packet cache; Real-time monitoring of the number of binary data packets accumulated in the data packet buffer and the channel utilization of multiple data parsing channels; If the number of data cascades exceeds the upper limit and the channel utilization rate remains higher than the high load threshold, then the number of data parsing channels will be increased. If the number of stacked data channels is lower than the minimum stacked data channel limit and the channel utilization rate remains below the low load threshold, then the number of data parsing channels will be reduced.

[0153] In some embodiments, each binary data packet contains at least one binary data, and each binary data contains second metadata and business fields. The parsing module is specifically used for: Based on the target data parsing channel and in accordance with the target parsing rules of the binary data packet, the business field and the second metadata corresponding to the business field are extracted from the binary data packet; The business field is converted into a target business field with a set format, and a third metadata is generated based on the target business field and the second metadata to obtain the structured parsed data containing the target business field and the third metadata; the third metadata contains the data related to the target business field in the second metadata and the corresponding newly added description data when generating the target business field.

[0154] In some embodiments, the sending module is specifically used for: Each piece of structured parsed data is formatted to obtain the data to be sent; According to the data type and / or service terminal identifier of each of the data to be sent, the multiple data to be sent are merged into at least one data packet to be sent, and the routing identifier of the target service terminal is configured for each data packet to be sent; Based on the routing identifier, the data packets to be sent are sent in parallel to the corresponding target service terminals.

[0155] In some embodiments, the system further includes an idempotency identifier processing module, configured to: Based on the device identifier, timestamp, and sequence number of each data packet to be sent, an idempotent identifier is generated for the data packet to be sent. Bind the idempotency identifier to the data packet to be sent; The data packet to be sent, which is bound to the idempotency identifier, is sent to the target service terminal so that the target service terminal can check the service processing status of the data packet to be sent based on the idempotency identifier, and perform service processing on the data packet to be sent if the service processing status is unprocessed.

[0156] In some embodiments, the system further includes a rule management module, used for: If the target parsing rule comes from a remote rule base, the target parsing rule is stored in the local rule base.

[0157] The data processing system provided in this application embodiment can implement each process of the above-described data processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0158] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. As shown in the figure, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown in the diagram), memory 41, and computer program 42 stored in said memory 41 and executable on said at least one processor 40, which, when executed, implements the steps in any of the above method embodiments.

[0159] The electronic device 4 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The electronic device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0160] The processor 40 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0161] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 41 can include both internal and external storage units of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0163] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0164] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] In the embodiments provided in this application, it should be understood that the disclosed systems / electronic devices and methods can be implemented in other ways. For example, the system / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of systems or units may be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0168] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0169] The processes in the above-described embodiments can be implemented by a computer program product. When the computer program product is run on an electronic device, the electronic device executes the steps in the above-described method embodiments.

[0170] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Receive multiple binary data packets transmitted from multiple terminal devices; Based on the first metadata of each binary data packet and the load status of multiple data parsing channels, each binary data packet is assigned to a target data parsing channel; The target data parsing channel is controlled to parse the binary data packet according to the target parsing rule of each binary data packet, so as to obtain at least one structured parsed data of each binary data packet; Each of the structured parsed data is sent to one or more target service terminals.

2. The method according to claim 1, characterized in that, The process of allocating each binary data packet to a target data parsing channel based on the first metadata of each binary data packet and the load status of multiple data parsing channels includes: Based on the device identifier, data type, and / or data priority in the first metadata of each binary data packet, the plurality of binary data packets are divided into at least one group to be assigned; From the plurality of data parsing channels, determine the target data parsing channel whose load status satisfies the load requirements of the group to be allocated for each of the multiple data parsing channels; The binary data packets in the group to be assigned are assigned to the matching target data parsing channel.

3. The method according to claim 1, characterized in that, After receiving multiple binary data packets transmitted by multiple terminal devices, the method further includes: Store multiple binary data packets in a data packet cache; The method further includes: Real-time monitoring of the number of binary data packets accumulated in the data packet buffer and the channel utilization of multiple data parsing channels; If the number of data cascades exceeds the upper limit and the channel utilization rate remains higher than the high load threshold, then the number of data parsing channels will be increased. If the number of stacked data channels is lower than the minimum stacked data channel limit and the channel utilization rate remains below the low load threshold, then the number of data parsing channels will be reduced.

4. The method according to claim 1, characterized in that, Each binary data packet contains at least one binary data item, and each binary data item contains second metadata and business fields. The control of the target data parsing channel parses the binary data packet according to the target parsing rules of each binary data packet to obtain at least one structured parsed data item for each binary data packet, including: Based on the target data parsing channel and in accordance with the target parsing rules of the binary data packet, the business field and the second metadata corresponding to the business field are extracted from the binary data packet; The business field is converted into a target business field with a set format, and a third metadata is generated based on the target business field and the second metadata to obtain the structured parsed data containing the target business field and the third metadata; the third metadata contains the data related to the target business field in the second metadata and the corresponding newly added description data when generating the target business field.

5. The method according to claim 1, characterized in that, Sending each of the structured parsed data to one or more target service terminals includes: Each piece of structured parsed data is formatted to obtain the data to be sent; According to the data type and / or service terminal identifier of each of the data to be sent, the multiple data to be sent are merged into at least one data packet to be sent, and the routing identifier of the target service terminal is configured for each data packet to be sent; Based on the routing identifier, the data packets to be sent are sent in parallel to the corresponding target service terminals.

6. The method according to claim 5, characterized in that, The method further includes: Based on the device identifier, timestamp, and sequence number of each data packet to be sent, an idempotent identifier is generated for the data packet to be sent. Bind the idempotency identifier to the data packet to be sent; The data packet to be sent, which is bound to the idempotency identifier, is sent to the target service terminal so that the target service terminal can check the service processing status of the data packet to be sent based on the idempotency identifier, and perform service processing on the data packet to be sent if the service processing status is unprocessed.

7. The method according to claim 1, characterized in that, If the target parsing rule comes from a remote rule base, the target parsing rule is stored in the local rule base.

8. A data processing system, characterized in that, include: The receiving module is used to receive multiple binary data packets transmitted from multiple terminal devices; The allocation module is used to allocate each binary data packet to a target data parsing channel based on the first metadata of each binary data packet and the load status of multiple data parsing channels; The parsing module is used to control the target data parsing channel to parse the binary data packet according to the target parsing rules of each binary data packet, so as to obtain at least one structured parsed data of each binary data packet; The sending module is used to send each of the structured parsed data to one or more target service terminals.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device performs the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 7 to be performed.

Images