Data transmission method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a data transmission method and device, electronic equipment and a readable storage medium, and the method comprises the steps: extracting the attribute information of Internet of Things equipment from a connection request sent by the Internet of Things equipment; according to the attribute information and the load condition of each proxy node, allocating the Internet of Things equipment to a first message proxy node corresponding to a first equipment group; writing first data sent by the Internet of Things equipment into a message queue through the first message proxy node; receiving a subscription request sent by the server, and writing the subscription request into a data processing rule list corresponding to the server; and processing the data in the message queue according to the data processing rule list corresponding to the server, and sending the processed data to the server, thereby improving the data transmission efficiency and improving the network bandwidth utilization rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data transmission method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] With the development of communication technology, the Internet of Things (IoT) is developing rapidly. The Internet of Things platform can realize the integration of "management, control and operation" of the underlying terminal device, provide application development and unified interface for the upper layer, and build an end-to-end channel for terminal devices and services. With the further maturity of artificial intelligence, machine learning and other technologies, the Internet of Things platform provides more powerful business analysis functions for users. With the rapid development of mobile Internet and the Internet of Things, the way to obtain information is becoming more and more diverse, from active information acquisition to passive information acceptance, and the amount of information is also showing an explosive growth.

[0003] MQTT (Message Queuing Telemetry Transport) is an instant messaging protocol developed by IBM. In the prior art, MQTT is widely used as a protocol for pushing messages from Internet of Things devices to the server. Since Internet of Things devices need to complete long-distance connection with the center server through the MQTT protocol, the reliability of MQTT transmission is low, and when there are many devices, data congestion will be faced. SUMMARY

[0004] The embodiments of the present application provide a data transmission method, device, electronic equipment and readable storage medium, which can improve the data transmission efficiency and improve the network bandwidth utilization.

[0005] In a first aspect, the embodiments of the present application disclose a data transmission method, which comprises: extracting attribute information of the Internet of Things device from a connection request sent by the Internet of Things device; According to the attribute information and the load condition of each proxy node, the Internet of Things device is distributed to a first message proxy node corresponding to a first device group; each device group corresponds to at least two message proxy nodes; write first data sent by the Internet of Things device to a message queue through the first message proxy node; receive a subscription request sent by a service end and write the subscription request to a data processing rule list corresponding to the service end; According to the data processing rule list corresponding to the service end, the data in the message queue is processed, and the processed data is sent to the service end.

[0006] In a second aspect, the embodiments of the present application disclose a data transmission device, the device comprising: an extraction module configured to extract attribute information of the Internet of Things device from a connection request sent by the Internet of Things device; an allocation module configured to allocate the Internet of Things device to a first message broker node corresponding to a first device group according to the attribute information and a load condition of each proxy node; each device group corresponds to at least two message broker nodes; a writing module configured to write first data sent by the Internet of Things device to a message queue through the first message broker node; a receiving module configured to receive a subscription request sent by a server and write the subscription request to a data processing rule list corresponding to the server; a sending module configured to process data in the message queue according to the data processing rule list corresponding to the server and send the processed data to the server.

[0007] In a third aspect, the embodiments of the present application disclose an electronic device, the electronic device comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used for storing executable instructions, and the executable instructions make the processor execute the data transmission method as described above.

[0008] In a fourth aspect, the embodiments of the present application disclose a readable storage medium, when instructions or transactions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the data transmission method as described above.

[0009] The embodiments of the present application have the following advantages: Attribute information of the Internet of Things device is extracted from a connection request sent by the Internet of Things device; the Internet of Things device is distributed to a first message broker node corresponding to a first device group according to the attribute information and a load condition of each proxy node, and the device grouping is no longer dependent on a fixed rule, but is dynamically adjusted in combination with the device attribute and the real-time state of the message broker node, so as to ensure that the connection pressure can be horizontally expanded to all message broker nodes in the cluster when the load is high, thereby avoiding that the load of a single message broker node is too high, improving the expansibility, shortening the time of connecting the device to the message broker node, and effectively relieving the congestion problem when a large number of Internet of Things devices are accessed. The first data sent by the Internet of Things device is written into a message queue through the first message broker node; a subscription request sent by a service end is received and written into a data processing rule list corresponding to the service end. The data in the message queue is processed according to the data processing rule list corresponding to the service end, and the processed data is sent to the service end, so that each service end does not need to repeatedly transmit data when acquiring data, and through one-time data transmission, the data is distributed to different service ends according to the requirements corresponding to each service end, the service end does not need to repeatedly pull data and perform redundant processing, the delay of acquiring data is reduced, and the operation and maintenance cost of the multiple service end systems is also reduced. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0011] Figure 1 is a step flow chart of an embodiment of a data transmission method of the present application; Figure 2 is a step flow chart of another embodiment of a data transmission method of the present application; Figure 3 is a step flow chart of another embodiment of a data transmission method of the present application; Figure 4 is a structural block diagram of a data transmission device of the present application; Figure 5 is a structural block diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0012] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0013] The terms “first”, “second”, and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by “first”, “second”, etc. are generally a class and do not limit the number of objects, for example, the first object can be one or more. In addition, the term “and / or” in the specification and claims is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents an “or” relationship between the associated objects. The term “multiple” in the embodiments of the present application means two or more, and other quantifiers are similar.

[0014] Before the present application is described, the application scenario of the present application is first described: With the rapid development of the Internet of Things industry, the scale of Internet of Things devices connected to the network grows exponentially, and the types of devices cover low-power embedded devices, medium-high-power gateway devices, and data transmission requirements present high-concurrent connection, multi-type data, low-latency, and low-power consumption characteristics. In the current Internet of Things data transmission field, two technical solutions are mainly used to realize the flow of device data to downstream business systems, but both have significant technical defects and are difficult to adapt to large-scale, high-reliability Internet of Things application scenarios.

[0015] A technical solution is to balance the lightweight connection characteristics of the MQTT protocol and the high-throughput persistence characteristics of the Apache Kafka Distributed Streaming Platform (Kafka). Some solutions adopt a device-side double-layer protocol integration mode. Specifically, the Internet of Things device first completes data collection through the MQTT protocol, then integrates a Kafka client in the device or edge gateway, converts the MQTT protocol encapsulated message into a Kafka compatible format, and finally writes the data into the Kafka cluster through the Kafka client. The downstream business system pulls data from the cluster through a Kafka consumer to complete subsequent processing. The MQTT protocol is a lightweight publish-subscribe message transmission protocol that uses Transmission Control Protocol (TCP) connections, with a message header of only 2 bytes. It is suitable for low-bandwidth and low-computing-power devices, and its core function is to achieve low-cost connection between devices and networks. Kafka is a distributed message queue system that implements high-throughput and persistent storage based on a Topic-Partition architecture, and its core function is to provide high-reliability data flow buffering for downstream systems. Kafka client logic refers to the Kafka interaction module that needs to be integrated in the device or gateway.

[0016] To simplify the complexity of the device side, some solutions adopt a centralized proxy mode, that is, a separate MQTT consumer proxy node (Broker) is deployed, and all Internet of Things devices establish a long connection with the Broker through the MQTT protocol and send the collected messages to the Broker. After receiving the messages, the Broker does not do any filtering or shunting processing, but directly pushes the full amount of messages to the downstream business system through a unified forwarding channel, or writes them to a single Kafka Topic. The downstream system needs to filter the data that meets its own needs from the full amount of messages. The connection number and message processing capacity of a single MQTT Broker have a physical upper limit. When the scale of Internet of Things devices exceeds the threshold, problems such as connection timeout, message accumulation, and forwarding delay may occur, which cannot support large-scale Internet of Things scenarios.

[0017] Method embodiment Reference Figure 1 A step flowchart of an embodiment of a data transmission method of the application is shown, and the method can specifically include the following steps: Step 101, extracting attribute information of the Internet of Things device from a connection request sent by the Internet of Things device; Step 102: Based on the attribute information and the load status of each proxy node, assign the IoT devices to the first message proxy nodes corresponding to the first device group; each device group corresponds to at least two message proxy nodes; Step 103: Write the first data sent by the IoT device into the message queue through the first message broker node; Step 104: Receive the subscription request sent by the server and write the subscription request into the data processing rule list corresponding to the server; Step 105: Process the data in the message queue according to the data processing rule list corresponding to the server, and send the processed data to the server.

[0018] Regarding steps 101 to 105, the data transmission method provided in this application embodiment can be applied to IoT platforms or IoT systems, such as IoT data platforms built by manufacturing enterprises, energy companies, and smart city projects; or edge computing frameworks, that is, achieving lightweight device access and data preprocessing at the edge. Furthermore, the data transmission method provided in this application embodiment can be applied to smart home scenarios, smart factories, smart cities, and connected vehicles. For example, in a smart home scenario, various home appliances and sensors are connected to a unified IoT platform, which distributes data to applications and data analysis systems; in a smart factory, machine tools and sensors are connected to an IoT platform, which distributes data to production execution systems and predictive maintenance platforms; and in a connected vehicle application scenario, real-time vehicle data is connected to a unified IoT platform and distributed to fleet management systems, vehicle owner mobile terminals, and traffic dispatch centers.

[0019] In this context, IoT devices refer to physical objects equipped with sensors and actuators and capable of connecting to the internet, such as smart meters, industrial machine tools, smart bracelets, and environmental temperature and humidity sensors. A connection request refers to the initialization message sent when a device establishes a communication session with the platform. Attribute information refers to metadata describing the characteristics or status of the device, such as device identifier, region, device type, model, and geographical location. Load status refers to the current resource consumption and workload indicators of the message broker node. A message broker node is an independent hardware or virtual carrier that runs the message broker service and undertakes message relay and connection management functions. In this application, a platform may consist of a cluster of multiple message broker nodes. A message queue refers to a persistent message storage system that acts as an intermediate buffer to decouple producers and consumers. For example, in Apache Kafka, a message queue refers to a raw data topic in Kafka. The server refers to the downstream business system, which is the final consumer of the processed data. Examples include a real-time monitoring and alarm system, a big data analytics platform, a data warehouse, and the backend of a user application. A subscription request refers to the server's requirement specification for the data it needs, submitted to the platform. For example, a JavaScript Object Notation (JSON) configuration might contain: {"system": "alarm_svc", "filter": "temperature>80"}, indicating that the alarm service requires data with a temperature greater than 80 degrees Celsius. A data processing rule list refers to the collection of data processing rules maintained for each server, compiled from its subscription requests. It's a list stored in a database or memory, containing multiple rules. For example, the rule maintained for the alarm service might be: [Rule1:if deviceType=='temp' and value>80 then formatToJSON() and routeTo('alarm-topic')].

[0020] IoT devices initiate connections using the MQTT protocol. Device attributes are extracted from the connection request, and combined with the current load of each message broker node, the device group (shared subscription group) to which it belongs is determined. The principle of grouping is to achieve load balancing across all message broker nodes as much as possible. Ensuring load balancing across multiple message broker nodes prevents a single message broker node from being overwhelmed.

[0021] After establishing a stable connection with the message broker node, IoT devices continuously report data (e.g., temperature data). Upon receiving this data, the message broker node does not process business logic but instead publishes it to a unified message queue, such as a Kafka Topic. All device data, regardless of which message broker node it originates from, is aggregated into the same Kafka Topic, achieving a unified data entry point and preparing for subsequent processing. Downstream servers submit subscription requests through the platform's Application Programming Interface (API). The platform adds the subscription request to the data processing rule list maintained for that server. Through on-demand subscription, downstream systems inform the platform of their data needs, rather than sifting through massive amounts of data themselves. The platform's stream processing engine monitors changes to the data processing rule list and continuously consumes raw data from the message queue. For each piece of raw data, it loads and processes the rule lists from all servers. For subscription requests from different servers, i.e., different downstream business systems, corresponding topics are created. The stream processing engine reads data from the message queue, i.e., the raw Kafka Topic, only once, yet can simultaneously satisfy the needs of multiple servers. The processed data is written to different, dedicated Kafka topics, and each server subscribes to only the topics it needs. Data is distributed independently to each server; for example, the big data platform consumes the `data_for_bigdata` topic, and the alarm service consumes the `data_for_alarm` topic. The two are completely isolated and do not interfere with each other.

[0022] In this embodiment, the challenge of handling massive numbers of devices with high concurrency is solved by device grouping and proxy node load balancing, avoiding the risk of congestion when a large number of devices access the platform. Device access is achieved through MQTT, and data buffering and decoupling are achieved through Kafka. Through a rule engine and stream processing, the raw data read at once is filtered into multiple customized data streams and distributed to different downstream systems, fundamentally avoiding duplicate data transmission and redundant processing, and greatly improving the overall efficiency and flexibility of the system.

[0023] Reference Figure 2 The flowchart illustrates another embodiment of the data transmission method of the present invention. The method may specifically include the following steps: Step 201: Extract the attribute information of the IoT device from the connection request sent by the IoT device.

[0024] Step 201 can be referred to in detail in step 101 above, and will not be repeated here.

[0025] Step 202: Determine the first device group based on the attribute information of the IoT devices; Step 203: Determine the mapping relationship between the first device group and each message broker node based on the load status of each broker node; Step 204: Determine the first message broker node from each message broker node corresponding to the first device group according to the preset algorithm.

[0026] For steps 202 to 204, the system extracts predefined attribute information of IoT devices from the device connection requests, and then maps these attributes to a logical device group according to a preset grouping strategy. Key attribute fields are parsed from the device's connection request certificate or message. Common attributes include: device type, region, project identifier, device level, etc. For example, the grouping strategy can be based on device type, region, or device level, or multiple attributes simultaneously. A unique device group identifier is assigned to each IoT device, achieving business isolation through grouping. Grouping devices with similar characteristics or business needs into the same group facilitates unified management, security policy settings, and data routing. For example, sensor data from all factories A can be aggregated for consumption by factory A's monitoring system.

[0027] The mapping between the first device group and each message broker node is not static but dynamically adjusted based on load. The system continuously monitors the real-time load of all message broker nodes and maintains a dynamic mapping table from device groups to a list of available broker nodes. Load metric collection refers to a separate monitoring module periodically collecting load metrics from each message broker node. Key metrics include: current connection count, processor and memory utilization, message throughput, network connection count, and system connection capacity. The load balancer or cluster management module receives these metrics, sets a load threshold, and uses these metrics to determine which message broker nodes are healthy and idle, and which are under excessive pressure. Then, based on the status of each message broker node, it dynamically updates a "group-process" mapping table. For example, the initial mapping is the list of broker nodes corresponding to the group `group_factory_a_sensors` [broker-1, broker-2, broker-3]. If the processor utilization of broker-2 consistently exceeds the threshold, the management module will temporarily remove it from the list of available processes in `group_factory_a_sensors`.

[0028] Once a device group is identified for an IoT device and a list of message broker nodes corresponding to that group is available, a simple algorithm is used to make the final selection from the list. Common algorithms include: round-robin, least connections, weighted round-robin, and consistent hashing. Round-robin selects the next node in turn, ensuring absolute even distribution. Least connections checks the current connection count of each message broker node in the list and selects the one with the fewest connections. Weighted round-robin assigns higher weights to more powerful message broker nodes, increasing their chances of being selected. Consistent hashing performs a hash calculation based on the device ID, ensuring that the same device always connects to the same broker node, suitable for stateful scenarios. By making the final selection from the message broker nodes corresponding to the device group, the device load is evenly distributed across all available message broker nodes within the group, achieving micro-level load balancing.

[0029] Optionally, step 203 may specifically include the following steps: Sub-step 2031: For each message broker node, if the load of the message broker node exceeds a preset threshold, migrate the device group corresponding to the message broker node to the second message broker node; the load of the second message broker node is lower than that of the message broker node.

[0030] In other words, when the system detects that a message broker node is overloaded, it automatically migrates a portion of the load it handles—that is, the group of devices it supports—to a less overloaded message broker node. Specifically, a monitoring agent continuously collects load metrics for all message broker nodes. The decision engine compares these metrics with preset thresholds. When it finds that a message broker node's metrics have consistently exceeded the threshold for a period of time, the decision engine triggers a rebalancing operation. The system filters candidate message broker nodes with lighter loads from the entire message broker node cluster, and then, based on a selection strategy (e.g., selecting the one with the lowest load), determines a target node from the candidates as the second message broker node.

[0031] Optionally, step 204 may specifically include the following steps: Sub-step 2041: Perform a hash calculation on the unique identifier of the IoT device to obtain a first hash value; Sub-step 2042: Based on the number of message broker nodes corresponding to the first device group, perform a modulo operation on the first hash value to obtain the index value; Sub-step 2043: Based on the index value, determine the first message broker node from each message broker node corresponding to the device group.

[0032] For sub-steps 2041 to 2043, the unique identifier of an IoT device refers to a string, number, or code that uniquely identifies the IoT device, such as the device's factory serial number or a custom unique identifier, ensuring that each device has an independent identifier in the system and avoiding confusion with other devices. For example, "sensor_001_room123" represents sensor number 001 in room 123, and "IMEI_861234567890123" is the device's International Mobile Equipment Identity. Hash calculation refers to the process of calculating a fixed-length value by using a hash function on the input unique identifier. The core characteristics of a hash function are that the same input will always produce the same output; different inputs are likely to produce different outputs; and the output values ​​are evenly distributed (mapping any input to a fixed range of values ​​with no obvious clustering). Its purpose here is to convert the string or encoded identifier into a computable value, facilitating subsequent allocation logic processing. Modulo operation refers to the mathematical operation of dividing the first hash value by the number of message broker nodes corresponding to the first device group and obtaining the remainder. The core function is to restrict the first hash value to a range of 0 to the number of proxy nodes - 1, ensuring that the result directly corresponds to a specific proxy node. The index value refers to the result of the modulo operation, which is a non-negative integer ranging from 0 to the number of proxy nodes - 1. Its function is to serve as an index in the message proxy node list, directly locating the specific proxy node. For example, an index value of 0 corresponds to the first proxy node in the list, an index value of 1 corresponds to the second proxy, and so on.

[0033] Assumptions: The first device group has three message broker nodes: Agent A, Agent B, and Agent C, listed in the order [Agent A (index 0), Agent B (index 1), Agent C (index 2)]. There are three IoT devices with unique identifiers "device_001", "device_002", and "device_003". A CRC32 hash function is used to hash the identifiers "device_001", "device_002", and "device_003", yielding three first hash values. Since the first device group has three message broker nodes, these three first hash values ​​are divided by 3 to obtain index values. Matching is then performed from the list based on these index values: index 0 points to Agent A, index 1 points to Agent B, and index 2 points to Agent C.

[0034] Step 205: Receive the subscription request sent by the server and write the subscription request into the data processing rule list corresponding to the server.

[0035] Optionally, step 205 may specifically include the following steps: Step S11: Upon receiving the subscription request, query whether a corresponding list of data processing rules exists based on the server identifier; Step S12: If the data processing rule list does not exist, create a corresponding data processing rule list for the server. Step S13: If the data processing rule list exists, add the subscription request to the data processing rule list.

[0036] For steps S11 to S13, by receiving subscription requests from the server, a dedicated data processing rule list, or consumption rule set, is maintained for each server. This ensures that the subscription needs of downstream systems are recorded and updated in real time, providing a basis for generating subsequent data distribution strategies. A subscription request is a data request instruction sent by the server, containing specific subscription parameters, such as: "requires sensor data with temperature > 30℃", "data format is JSON", "Quality of Service (QoS) level is 1", "consumption rate does not exceed 100 messages / second", etc. Server identifier: A unique identifier for the server, such as "monitor_system_001" or "analysis_platform_002", used to distinguish the subscription needs of different servers and avoid rule confusion. The data processing rule list is essentially a set of subscription needs maintained for each server, storing all subscription rules for that server in a structured form, such as lists or key-value pairs. This data processing rule list supports dynamic addition and modification and serves as the basis for subsequent data filtering and distribution.

[0037] The system receives a subscription request from the server, extracts the server identifier from the request, and uses this identifier as the query key to search the storage area of ​​the consumption rule topic to see if a list of data processing rules corresponding to that server already exists. It then determines whether this is the server's first subscription, a supplementary subscription, or a modification subscription, providing a basis for subsequent operations.

[0038] If the query result is no matching rule list (meaning this is the server sending a subscription request for the first time), initialize a new data processing rule list using the server identifier as the unique key. Store the newly created rule list in the storage area of ​​the consumption rule topic, such as writing it to a Kafka consumption-rule-topic or storing it in a distributed database. When initializing the list, use the currently received subscription request as the first rule in the list; for example, the rule list is initialized as [{"filter condition":"temperature>30", "data format":"JSON", "QoS":1}]. Create a dedicated rule storage container for the new server to ensure that its subscription request is officially recorded.

[0039] If the query result shows a matching rule list, meaning the server has previously sent a subscription request and has existing historical rules, the existing data processing rule list of the server is read from the storage area. Newly received subscription requests are validated. If they conflict with existing rules, and not with historical rules, they are appended directly. If the new rule conflicts with the data format of historical rules, a preset strategy is used, such as overwriting the old rule or retaining both. The validated new subscription request is then added to the rule list. For example, the updated list might be [{"filter condition":"temperature>30", "data format": "JSON"}, {"filter condition":"humidity>60", "data format":"Protobuf"}]). The updated rule list is written back to the storage area, overwriting the old version, to ensure the data in the consumption rule topic is up-to-date. This application supports dynamic expansion of subscription requirements by the server, enabling one-time subscription and multiple replenishments, avoiding resource waste caused by repeatedly creating lists.

[0040] Optionally, step 205 may specifically include the following steps: Step S21: Convert the subscription request into a format that can be executed by data processing rules; Step S22: Write the data processing rules into the data processing rule list corresponding to the server.

[0041] Regarding steps S21 and S22, in this embodiment, the subscription request is standardized and persisted. The core idea is to convert the original subscription request sent by the server into rules that the system can directly execute, and securely store them in the corresponding data processing rule list, providing an executable basis for subsequent data distribution. Specifically, the original subscription request is a non-standardized requirement expression sent by the server, which may be API parameters, JSON strings, or even semi-structured text, such as: "Give me data for devices of type air conditioner with a temperature > 30°C, aggregated by hour, in JSON format." The format may vary depending on the server-side development language and interface design. The executable data processing rules refer to the standardized structured rules defined by the system. The rules contain explicit fields, logical operators, and execution instructions (e.g., {"filter": {"device_type":"air conditioner","temperature":{"gt":30}},"aggregate":{"interval":"1h"},"output_format":"JSON"}), which can be directly parsed and executed by the stream processing engine in the system.

[0042] Specifically, the subscription request is parsed, and the validity of each field is verified. Finally, the valid information after parsing the subscription request is mapped to the system's predefined rule template, unifying the field names, data types, and logical expressions.

[0043] Step 206: Process the data in the message queue according to the data processing rule list corresponding to the server, and send the processed data to the server.

[0044] Optionally, step 206 may specifically include the following steps: Step S31: Generate a corresponding data processing task for each data processing rule in the data processing rule list; Step S32: Obtain data from the message queue by executing the data processing task, process the data by applying data processing rules, and send the processed data to the server.

[0045] Regarding steps S31 and S32, the data processing rule list refers to the set of all subscription requests corresponding to the server, such as "sensor data with temperature > 30℃" or "hourly aggregated device status data," etc. Each rule includes information such as filtering conditions, processing logic, and output format. A data processing task refers to a specific execution unit generated based on a single data processing rule, containing information such as the complete logic of the rule, data source, and execution resources; it is the smallest unit actually scheduled by the system.

[0046] Message queues are distributed queues that store raw or pre-processed data from IoT devices. Data is continuously written in the form of a stream for downstream tasks to consume.

[0047] Specifically, the system periodically or in real-time monitors the server's data processing rule list. When a new, updated, or deleted rule is detected, all valid rules in the list are processed sequentially, filtering out expired or disabled rules. Example: Server A's rule list contains two valid rules: rule 1 is "temperature > 30℃" and rule 2 is "humidity > 60%". The system then iterates through these two rules to generate tasks for each. For each rule, core information is extracted, and necessary execution parameters are added to generate the task's metadata, such as the task's identifier and execution description. The task identifier uniquely identifies the task, such as "task_rule1_serverA", and is generated by combining the rule identifier and the server identifier. It is used for task scheduling and status tracking. The associated rule identifier binds to the corresponding rule, such as "rule_001", ensuring the correspondence between tasks and rules and facilitating synchronized task adjustments during subsequent rule updates. The data source specifies the data source to be consumed in the message queue, such as a raw-sensor-data topic in Kafka or a specific partition. Resources are allocated based on rule complexity: simple rules receive lighter resources, while complex rules receive more resources. The scheduling strategy defines the task execution method; for example, real-time stream processing means processing data immediately after it is written, while batch processing can process accumulated data every 5 minutes. The filtering, processing, and output logic in the rules is transformed into executable code snippets or configuration instructions. Filtering logic can be transformed into a conditional expression like "temperature > 30℃" using `WHERE temperature > 30`. Processing logic can be transformed into an aggregation instruction like `GROUP BY TUMBLE(time, INTERVAL '1' HOUR) SELECT AVG(temperature)`. Output logic can be transformed into an output instruction like "JSON format, push to server A's API" using `FORMAT AS JSON; SEND TO http: / / serverA:8080 / data`. By executing the code snippets or configuration instructions, the data is processed and distributed to the servers corresponding to the subscription requirements.

[0048] Optionally, step 206 may specifically include the following steps: Step S41: Create a corresponding data distribution channel for the server; the data distribution channel is used to transmit data that conforms to each data processing rule in the data processing rule list; Step S42: Write the processed data into the data distribution channel so that the server can obtain data from the data distribution channel through the corresponding set of consumption units; wherein, the set of consumption units includes at least one consumption unit, which is used to consume data from the data distribution channel.

[0049] Regarding steps S41 and S42, the data distribution channel refers to the logical data flow pipeline created for the server, which only transmits data that conforms to the server's data processing rule list. For example, there might be a data channel that only transmits sensor data with a temperature greater than 30°C, achieving physical isolation between data from different servers. Technically, this can be achieved using Kafka Topics, dedicated topic branches in MQTT, or independent message queues, providing features such as rule-based data filtering, independent storage, and dedicated access. The consumer unit set refers to the data consumption execution group deployed on the platform, consisting of at least one consumer unit, such as a Kafka Consumer instance and a middleware process. Each consumer unit is responsible for pulling data from the data distribution channel and delivering it to the server, supporting parallel processing to improve efficiency.

[0050] Specifically, the system reads the server's data processing rule list and extracts all rule characteristics, such as data type, common filtering conditions, and estimated data volume. Example: Server C's rule list contains two rules: "Temperature > 30℃" and "Humidity > 60%", both targeting workshop sensors. Analysis indicates that this channel needs to transmit high-temperature or high-humidity data from the workshop sensors, with an estimated data volume of 500 records / second. Based on the rule analysis results, specific attributes are set for the channel to ensure it matches the server's requirements. Examples include: channel identifier, data filtering benchmark, storage strategy, and partition configuration. The system creates a channel and associates it with data processing rules; it establishes a binding relationship between the channel and the server's data processing rule list, ensuring that only data matching the rules in that list will enter the channel. Example: A Kafka Topic is created as "channel_serverC_workshop_sensor", configured with two partitions, data retention for 24 hours, and bound to the server C's rule list ID "rule_list_C_001". Register the channel metadata and notify the relevant modules to register the channel's attributes to the channel metadata center so that the system can query the channel information; notify the data processing task module that a new channel has been created, and subsequent processed data that conforms to the server rules must be written to this channel.

[0051] It should be noted that each data processing rule list, also known as a consumption rule list, corresponds to a server. Since a server may have multiple business scenario requirements, multiple consumption rules (i.e., data processing rules) will be generated. Therefore, each data processing list includes at least one data processing rule. In this embodiment, only one full data retrieval is needed. By uniformly filtering and processing the data according to each consumption rule, a corresponding consumption rule topic is set for each server. The data required by each server is stored in the corresponding consumption rule topic. Each server only needs to subscribe to the corresponding consumption rule topic to obtain the data that meets its requirements, without needing to retrieve all the raw data itself. Data retrieval between each server is independent and does not interfere with each other, thereby reducing the bandwidth occupied by the platform when sending data to various servers, reducing bandwidth consumption, and improving network bandwidth utilization.

[0052] A dedicated set of consumer units, or consumer groups, is created for each channel on the server side. These consumer groups are then bound to channels to ensure that only that consumer group can consume the corresponding channel. In other words, based on the subscription requirements of each downstream system, corresponding consumption rules (i.e., data processing rules) are generated. Physical channels are created and isolated according to these data processing rules, and corresponding data distribution strategies are generated. The raw data is then processed according to these strategies, and the consumer groups obtain data that meets their subscription requirements and consumption rules through the channels. Data is pulled by the consumer groups and delivered in a manner supported by the server—that is, actively pushed—by using interfaces provided by the server to push data to the server's receiving address. Alternatively, data can be passively cached, writing it to a shared cache accessible to the server and notifying the server via callbacks that the data is ready, allowing the server to actively pull it from the cache.

[0053] In summary, the data transmission method provided in this application can extract the attribute information of IoT devices from connection requests sent by IoT devices. Based on the attribute information and the load of each proxy node, IoT devices are assigned to the first message proxy node corresponding to the first device group. Device grouping no longer relies on fixed rules but is dynamically adjusted based on device attributes and the real-time status of the message proxy nodes. This ensures that connection pressure under high load can be horizontally scaled to all message proxy nodes in the cluster, thereby avoiding excessive load on a single message proxy node, improving scalability, shortening the time for devices to connect to message proxy nodes, and effectively alleviating congestion problems during large-scale IoT device access. The first message proxy node writes the first data sent by the IoT device into a message queue; it receives subscription requests from the server and writes the subscription requests into the data processing rule list corresponding to the server. According to the data processing rule list corresponding to the server, the data in the message queue is processed, and the processed data is sent to the server. This eliminates the need for repeated data transmission when each server obtains data. Through a single data transmission, data is distributed to different servers according to their respective needs. Servers do not need to repeatedly fetch data or perform redundant processing, reducing data acquisition latency and lowering the operation and maintenance costs of multiple server systems.

[0054] Reference Figure 3 The flowchart illustrates another embodiment of the data transmission method of the present invention. The method may specifically include the following steps: Step A11: The IoT device reports its location information; Step A12: The data transmission platform obtains the device area identifier through the device's geographical location and assigns a group; Step A13: Bind message broker nodes to data transmission platform groups; Step A14: The data transmission platform allocates a shared subscription to each group; Step A15: The original topics of the stream processing service in the data transmission platform are grouped and stored based on shared subscription groups; Step A16: For each shared subscription group, adapt and aggregate according to the rules of each group; Step A17: Split the data according to the group and write it into the business data topic of the stream processing platform; Step A18: The downstream system obtains data from the data transmission platform through the consumer group.

[0055] In this embodiment, IoT devices report their geographic location information to the data transmission platform. The platform parses the region identifier based on the device's geographic location and groups devices in the same region together. A corresponding message broker node is bound to each device group to enable independent processing of grouped data. A shared subscription mechanism is configured for each group, allowing multiple consumer units to share data within the group and avoiding duplicate consumption. The original data topics of the stream processing service are stored according to the shared subscription groups, ensuring independent management of data within each group. Data is processed according to preset rules for each shared subscription group. The processed data is split by device group and written to the business data topics of the stream processing platform. Downstream systems retrieve the processed data of their respective groups from the business data topics through the corresponding consumer groups.

[0056] In this embodiment, geographical location-based grouping makes data processing more aligned with regional business needs. Grouping and binding to independent proxy nodes avoids excessive pressure on single nodes; shared subscriptions reduce redundant data transmission and improve resource utilization. Group-based splitting of business data topics allows downstream systems to retrieve only data from their respective groups, reducing invalid data processing and lowering integration complexity. The grouping architecture supports adding new regions; when adding devices, only the resources of the corresponding group need to be expanded, without adjusting the overall system, thus adapting to large-scale device access scenarios.

[0057] Device Examples like Figure 4 As shown, Figure 4 The diagram shows a logic block diagram of a data transmission apparatus according to an embodiment of the present invention. The apparatus may include: Extraction module 410 is used to extract attribute information of the Internet of Things (IoT) device from the connection request sent by the IoT device; The extraction module 420 is used to allocate the IoT devices to the first message proxy nodes corresponding to the first device group based on the attribute information and the load status of each proxy node; each device group corresponds to at least two message proxy nodes; The writing module 430 is used to write the first data sent by the IoT device into the message queue through the first message broker node; The receiving module 440 is used to receive a subscription request sent by the server and write the subscription request into the data processing rule list corresponding to the server. The sending module 450 is used to process the data in the message queue according to the data processing rule list corresponding to the server, and send the processed data to the server.

[0058] Optionally, the allocation module includes: The device group determination module is used to determine a first device group based on the attribute information of the IoT devices; The mapping relationship determination module is used to determine the mapping relationship between the first device group and each message broker node based on the load of each broker node; The process determination module is used to determine the first message broker node from each message broker node corresponding to the first device group according to a preset algorithm.

[0059] Optionally, the process determination module includes: The hash calculation module is used to perform hash calculation on the unique identifier of the IoT device to obtain a first hash value; The modulo operation module is used to perform a modulo operation on the first hash value according to the number of message broker nodes corresponding to the first device group to obtain the index value; The process determination submodule is used to determine the first message broker node from each message broker node corresponding to the device group based on the index value.

[0060] Optionally, the mapping relationship determination module includes: The migration module is used to migrate the device group corresponding to each message broker node to a second message broker node when the load of the message broker node exceeds a preset threshold; the load of the second message broker node is lower than that of the message broker node.

[0061] Optionally, the sending module includes: The task generation module is used to generate corresponding data processing tasks for each data processing rule in the data processing rule list. The acquisition module is used to acquire data from the message queue by executing the data processing task, process the data by applying data processing rules, and send the processed data to the server.

[0062] Optionally, the sending module includes: The channel creation module is used to create a corresponding data distribution channel for the server; the data distribution channel is used to transmit data that conforms to each data processing rule in the data processing rule list. The first writing submodule is used to write the processed data into the data distribution channel, so that the server can obtain data from the data distribution channel through the corresponding set of consumption units; wherein, the set of consumption units includes at least one consumption unit, which is used to consume data from the data distribution channel.

[0063] Optionally, the writing module includes: The query module is used to query whether a corresponding list of data processing rules exists based on the server identifier when the subscription request is received. Create a submodule to create a corresponding data processing rule list for the server if the data processing rule list does not exist; An add module is used to add the subscription request to the data processing rule list if the data processing rule list exists.

[0064] Optionally, the writing module includes: The conversion module is used to convert the subscription request into a format that can be executed by the data processing rules. The second writing submodule is used to write the data processing rules into the data processing rule list corresponding to the server.

[0065] In summary, the data transmission device provided in this application can extract the attribute information of IoT devices from connection requests sent by IoT devices. Based on the attribute information and the load of each proxy node, the IoT devices are assigned to the first message proxy node corresponding to the first device group. Device grouping no longer relies on fixed rules but is dynamically adjusted based on device attributes and the real-time status of the message proxy nodes. This ensures that connection pressure under high load can be horizontally scaled to all message proxy nodes within the cluster, thereby avoiding excessive load on a single message proxy node, improving scalability, shortening the time for devices to connect to message proxy nodes, and effectively alleviating congestion problems during large-scale IoT device access. The first message proxy node writes the first data sent by the IoT device into a message queue; it receives subscription requests from the server and writes the subscription requests into the data processing rule list corresponding to the server. According to the data processing rule list corresponding to the server, the data in the message queue is processed, and the processed data is sent to the server. This eliminates the need for repeated data transmission when each server obtains data. Through a single data transmission, data is distributed to different servers according to their respective needs. Servers do not need to repeatedly fetch data or perform redundant processing, reducing data acquisition latency and lowering the operation and maintenance costs of multiple server systems.

[0066] The data transmission device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a GPU BOX, mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific device.

[0067] The data transmission device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0068] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described data transmission method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0069] In embodiments of this application, the memory 502 can be used to store software programs and various data. The memory 502 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, applications or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 502 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 502 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0070] Processor 501 may include one or more processing units; optionally, processor 501 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 501.

[0071] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data transmission method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0072] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0073] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes described in the above-described data transmission method embodiments, and can achieve the same technical effects. To avoid repetition, further details are omitted here.

[0074] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0076] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data transmission method, characterized in that, The method includes: Extract the attribute information of the IoT device from the connection request sent by the IoT device; Based on the attribute information and the load status of each proxy node, the IoT devices are assigned to the first message proxy nodes corresponding to the first device group; each device group corresponds to at least two message proxy nodes; The first data sent by the IoT device is written into the message queue through the first message broker node; Receive a subscription request sent by the server and write the subscription request into the data processing rule list corresponding to the server; The data in the message queue is processed according to the data processing rule list corresponding to the server, and the processed data is sent to the server.

2. The method according to claim 1, characterized in that, The step of allocating the IoT devices to the target message proxy nodes corresponding to the first device group based on the attribute information and the load status of each proxy node includes: Based on the attribute information of the IoT devices, a first device group is determined; Based on the load of each proxy node, determine the mapping relationship between the first device group and each message proxy node; According to a preset algorithm, the first message broker node is determined from each message broker node corresponding to the first device group.

3. The method according to claim 2, characterized in that, The step of determining the first message broker node from each message broker node corresponding to the first device group according to a preset algorithm includes: The unique identifier of the IoT device is hashed to obtain a first hash value; Based on the number of message broker nodes corresponding to the first device group, perform a modulo operation on the first hash value to obtain the index value; Based on the index value, the first message broker node is determined from each message broker node corresponding to the device group.

4. The method according to claim 2, characterized in that, Determining the mapping relationship between the first device group and each message broker node based on the load status of each broker node includes: For each message broker node, if the load of the message broker node exceeds a preset threshold, the device group corresponding to the message broker node is migrated to a second message broker node; the load of the second message broker node is lower than that of the message broker node.

5. The method according to claim 1, characterized in that, The step of processing the data in the message queue according to the data processing rule list corresponding to the server, and sending the processed data to the server, includes: For each data processing rule in the data processing rule list, a corresponding data processing task is generated; The data processing task retrieves data from the message queue, processes the data using data processing rules, and then sends the processed data to the server.

6. The method according to claim 1, characterized in that, The step of processing the data in the message queue according to the data processing rule list corresponding to the server, and sending the processed data to the server, includes: A corresponding data distribution channel is created for the server; the data distribution channel is used to transmit data that conforms to each data processing rule in the data processing rule list; The processed data is written into the data distribution channel so that the server can obtain data from the data distribution channel through the corresponding set of consumption units; wherein, the set of consumption units includes at least one consumption unit, which is used to consume data from the data distribution channel.

7. The method according to claim 1, characterized in that, The process of receiving a subscription request from the server and writing the subscription request into the data processing rule list corresponding to the server includes: Upon receiving the subscription request, query whether a corresponding list of data processing rules exists based on the server identifier; If the data processing rule list does not exist, a corresponding data processing rule list will be created for the server. If the data processing rule list exists, the subscription request will be added to the data processing rule list.

8. The method according to claim 1, characterized in that, The process of receiving a subscription request from the server and writing the subscription request into the data processing rule list corresponding to the server includes: The subscription request is formatted and converted to be executable data processing rules; The data processing rules are written into the data processing rule list corresponding to the server.

9. A data transmission device, characterized in that, The device includes: An extraction module is used to extract attribute information of the Internet of Things (IoT) device from the connection request sent by the IoT device. The allocation module is used to allocate the IoT devices to the first message proxy nodes corresponding to the first device group based on the attribute information and the load status of each proxy node; each device group corresponds to at least two message proxy nodes; The writing module is used to write the first data sent by the IoT device into the message queue through the first message broker node; The receiving module is used to receive subscription requests sent by the server and write the subscription requests into the data processing rule list corresponding to the server. The sending module is used to process the data in the message queue according to the data processing rule list corresponding to the server, and send the processed data to the server.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store executable instructions that cause the processor to perform a data transfer method as described in any one of 1 to 8.

11. A readable storage medium, characterized in that, When the instructions or transactions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data transmission method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes instructions or transactions that, when executed by a processor in an electronic device, cause the electronic device to perform the data transmission method of any one of claims 1 to 8.

Images