Lightweight data transfer method, system and apparatus for internet-of-things platform, and storage medium

Abstract

Disclosed in the present invention are a lightweight data transfer method, system and apparatus for an Internet-of-Things platform, and a storage medium. The method comprises: acquiring Internet-of-Things device data, and on the basis of a data format, writing the Internet-of-Things device data into corresponding topics of a message queue, and storing same; performing preprocessing on the Internet-of-Things device data in different topics, so as to obtain preprocessed Internet-of-Things device data; determining an operation result on the basis of a preset service processing flow and the preprocessed Internet-of-Things device data; performing standardization processing on the operation result on the basis of a destination type of the operation result, so as to obtain standardized data; and delivering the standardized data to a specified destination end, and acquiring a delivery result. The embodiments of the present invention reduce the hardware device cost and the maintenance cost, and improve the stability, and thus can be widely applied to the technical field of the Internet of Things.

Description

Lightweight data transfer method, system, device and storage medium for Internet of Things platform Technical Field

[0001] The present invention relates to the field of Internet of Things technology, and in particular to a lightweight data flow method, system, device and storage medium for an Internet of Things platform. Background Art

[0002] The IoT platform connects various devices, applications, and systems. The data exchange generated during communication is one of its core functions. Accurate and efficient data exchange ensures the platform's stability and intelligence. While many mature data management tools are currently available, they have significant shortcomings for smaller, resource-constrained platforms, particularly in the following areas: high hardware and maintenance costs, and poor stability. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present invention is to provide a lightweight data flow method, system, device and storage medium for an Internet of Things platform to reduce hardware equipment and maintenance costs and improve stability.

[0004] In one aspect, an embodiment of the present invention provides a lightweight data transfer method for an Internet of Things platform, comprising:

[0005] Obtain IoT device data and write the IoT device data into the topic corresponding to the message queue for storage according to the data format;

[0006] Preprocessing the IoT device data in different topics to obtain preprocessed IoT device data;

[0007] Determine the operation result according to the preset business processing flow and the pre-processed IoT device data;

[0008] Standardizing the operation result according to the destination type of the operation result to obtain standardized data;

[0009] Deliver the standardized data to the designated destination and obtain the delivery result.

[0010] Optionally, the preprocessing includes data format standardization conversion and / or key field extraction, and the preprocessing of the IoT device data in different topics specifically includes:

[0011] Adding source information to the IoT device data in different topics, the source information including the source topic and information identifier;

[0012] and / or,

[0013] Extract key fields from the IoT device data in different topics and add the key fields to the IoT device data.

[0014] Optionally, the Internet of Things platform lightweight data transfer method further includes:

[0015] Obtain the business processing flow of a specific business scenario through the business logic processing interface.

[0016] Optionally, the business processing flow includes several processing points, and the Internet of Things platform lightweight data flow method further includes:

[0017] When the business processing flow triggers a preset key processing point, the operation result of the key processing point is written into the key operation processing information topic of the message queue for storage.

[0018] Optionally, the Internet of Things platform lightweight data transfer method further includes:

[0019] The delivery result is written into the delivery result topic of the message queue for storage;

[0020] and / or,

[0021] The delivery results are added to the normalized data.

[0022] On the other hand, an embodiment of the present invention provides a lightweight data transfer system for an Internet of Things platform, including:

[0023] The first module is used to obtain IoT device data and write the IoT device data into the topic corresponding to the message queue for storage according to the data format;

[0024] The second module is used to preprocess the IoT device data in different topics to obtain preprocessed IoT device data;

[0025] The third module is used to determine the operation result according to the preset business processing flow and the pre-processed IoT device data;

[0026] A fourth module is configured to perform standardization processing on the operation result according to the destination type of the operation result to obtain standardized data;

[0027] The fifth module is used to deliver the standardized data to a designated destination and obtain a delivery result.

[0028] On the other hand, an embodiment of the present invention provides a lightweight data transfer device for an Internet of Things platform, comprising:

[0029] at least one processor;

[0030] at least one memory for storing at least one program;

[0031] When the at least one program is executed by the at least one processor, the at least one processor implements the lightweight data flow method for the Internet of Things platform as claimed in claim 1.

[0032] On the other hand, an embodiment of the present invention provides a computer-readable storage medium storing a program executable by a processor. When the program is executed by the processor, it is used for the above-mentioned lightweight data transfer method of the Internet of Things platform.

[0033] On the other hand, an embodiment of the present invention provides a lightweight data flow system for an Internet of Things platform, including an Internet of Things device, an Internet of Things platform, and a destination end, wherein the Internet of Things platform connects the Internet of Things device and the destination end;

[0034] The IoT device is used to collect IoT device data and send the IoT device data to the IoT platform;

[0035] The IoT platform is used to execute the aforementioned lightweight data transfer method for the IoT platform;

[0036] The destination end is used to receive the standardized data delivered by the Internet of Things platform and feedback the delivery result.

[0037] Optionally, the Internet of Things platform includes a data processing unit manager and several data processing units, one data processing unit corresponds to a topic in the message queue, and one data processing unit corresponds to several destination ends.

[0038] The implementation of the embodiment of the present invention includes the following beneficial effects: this embodiment obtains IoT device data and writes the IoT device data into the topic corresponding to the message queue according to the data format for storage; pre-processes the IoT device data in different topics to obtain pre-processed IoT device data; determines the operation result according to the preset business processing flow and the pre-processed IoT device data; standardizes the operation result according to the destination type of the operation result to obtain standardized data; delivers the standardized data to the specified destination end and obtains the delivery result; converts the IoT device data into a data stream according to the data format for storage. During the data flow process, different data formats are constructed according to the processing purpose to improve data processing efficiency. The business processing flow supports dynamic adjustment of resources to adapt to the real-time requirements of streaming data processing, make up for resource shortages, reduce costs and increase efficiency, and achieve timeliness, integrity and reliability of IoT data transmission; in addition, streaming data processing can be compatible with IoT platform scenarios of different data scales. While having efficient data flow processing capabilities, it streamlines hardware resources, saves memory consumption, and is better compatible with small-scale IoT scenarios, thereby reducing hardware equipment and maintenance costs and improving stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a structural block diagram of a lightweight data transfer system for an Internet of Things platform provided by an embodiment of the present invention;

[0040] FIG2 is a schematic diagram of a step flow diagram of a lightweight data transfer method for an Internet of Things platform provided by an embodiment of the present invention;

[0041] FIG3 is a schematic diagram of the steps of another method for lightweight data transfer on an Internet of Things platform provided by an embodiment of the present invention;

[0042] FIG4 is a schematic diagram of a flow chart of steps for preprocessing IoT device data in different topics according to an embodiment of the present invention;

[0043] 5 is a schematic diagram of a process flow for obtaining a business processing flow and storing operation results according to an embodiment of the present invention;

[0044] FIG6 is a schematic flow chart of steps for processing a delivery result according to an embodiment of the present invention;

[0045] 7 is a schematic diagram of a flow chart of the steps of a method for transferring data according to a specific embodiment of the present invention;

[0046] FIG8 is a structural block diagram of a lightweight data transfer system for an Internet of Things platform provided by an embodiment of the present invention;

[0047] 9 is a structural block diagram of a lightweight data transfer device for an Internet of Things platform provided by an embodiment of the present invention;

[0048] FIG10 is a structural block diagram of a lightweight data transfer system for an Internet of Things platform according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0050] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification and claims and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0052] Some technical terms in this embodiment are explained below.

[0053] MQ (Message Queue): A communication mechanism for asynchronous communication between applications. Applications write messages to a queue, and other applications read messages from it. Applications communicate through a shared message queue, rather than directly establishing dedicated connections. This communication model allows applications to continue executing after a message is sent, without waiting for a response from the recipient.

[0054] Pulsar: Apache Pulsar is a fast, scalable, and highly reliable distributed message queuing and stream processing platform. Pulsar supports multi-tenancy, multi-datacenter replication, and various messaging models. It offers high throughput and low latency while providing reliable messaging guarantees and supporting various message semantics.

[0055] Kafka: Apache Kafka is a distributed stream processing platform and message queue system focused on high-throughput messaging. Kafka's core concepts are topics and partitions. Data is divided into multiple partitions and distributedly stored to achieve high availability and scalability. Kafka provides persistent message storage and guarantees reliable message delivery, including replication to prevent data loss.

[0056] Nacos: Nacos is a dynamic service discovery, configuration management, and service management platform that makes it easier to build cloud-native applications. It serves as a service infrastructure for building modern service-centric application architectures (such as the microservices paradigm and the cloud-native paradigm), and is dedicated to helping discover, configure, and manage microservices. Nacos provides a set of easy-to-use features, including service discovery, dynamic configuration services, dynamic DNS services, and service and metadata management, to help quickly implement dynamic service discovery, service configuration, service metadata, and traffic management.

[0057] As shown in Figure 1, multiple IoT devices are connected to an IoT platform, which in turn connects to multiple destinations. IoT devices send collected data to the IoT platform, which then transmits the results of operations to the corresponding destinations. The IoT platform's processing includes data preprocessing, analysis, result processing, and result delivery.

[0058] 2 and 3 , an embodiment of the present invention provides a lightweight data transfer method for an Internet of Things platform, including steps S100 to S500 .

[0059] S100: Obtain IoT device data, and write the IoT device data into a topic corresponding to a message queue according to a data format for storage.

[0060] The IoT platform connects the IoT devices that need to be managed. Different IoT devices use different protocols. At the same time, when the IoT devices perform different operations, such as when the device goes online and offline, and when instructions are issued, the data formats are also different; that is, IoT devices can generate different data formats when using different protocols or performing different operations. The IoT device data is classified according to the corresponding data formats generated under different operations, and written into different topics of the message queue to facilitate subsequent conversion and parsing. At the same time, the protocol type used by various IoT devices will also be synchronously written into the topic as a tag, and will exist in the data flow of the entire process. The method in the embodiment of the present invention can simultaneously support the connection of different MQ middleware, such as Pulsar, Kafka, etc., to adapt to deployment under different resource conditions.

[0061] S200: Preprocess the IoT device data in different topics to obtain preprocessed IoT device data.

[0062] The specific processing method of pre-processing is determined according to the actual application and is not specifically limited in this embodiment. The pre-processed IoT device data is still stored according to the previous message queue topic.

[0063] Optionally, referring to FIG4 , preprocessing includes data format standardization conversion and / or key field extraction, and preprocessing IoT device data in different topics, specifically including:

[0064] S210. Add source information to IoT device data in different topics, where the source information includes a source topic and an information identifier.

[0065] and / or,

[0066] S220. Extract key fields from IoT device data in different topics, and add the key fields to the IoT device data.

[0067] Specifically, source information is added to collected IoT device data. This information includes the source topic and traceId. The traceId uniquely identifies the message and is used to track its flow within the platform. Key fields such as tenant information, device information, product information, and the type of device-triggered operation are extracted from the raw IoT device data to facilitate subsequent data analysis and processing. At the same time, the raw data collected from the IoT devices remains retained and participates in subsequent data flows.

[0068] It should be noted that step S210 and step S220 can be executed simultaneously, or only one of the steps can be executed.

[0069] S300: Determine the operation result according to the preset business processing flow and the pre-processed IoT device data.

[0070] Different business scenarios correspond to different business processes. The IoT platform can pre-define business processes for some basic services and can also obtain external input to map specific business processes to specific scenarios. Operation results represent all the results of data operations performed on pre-processed IoT device data according to the pre-determined business processes.

[0071] Optionally, referring to FIG5 , the lightweight data transfer method of the Internet of Things platform further includes:

[0072] S310: Obtain the business processing flow of a specific business scenario through the business logic processing interface.

[0073] The IoT platform provides a business logic processing interface. Through this interface, you can obtain the business process flow for specific business scenarios based on actual business scenario requirements and process and analyze IoT device data. This supports common data management scenarios such as data forwarding, rule matching, message push, alarm triggering, and scenario linkage. Furthermore, you can flexibly modify business logic to meet more customized scenarios.

[0074] Optionally, the business processing flow includes several processing points, and the lightweight data flow method of the Internet of Things platform further includes:

[0075] S320: When the business processing flow triggers a preset key processing point, the operation result of the key processing point is written into the key operation processing information topic of the message queue for storage.

[0076] When a business process triggers a key processing point, the operation results will be synchronously written into a dedicated key operation processing information topic for storage, which can track the current processing progress more quickly and avoid querying complex and redundant data.

[0077] S400: Standardize the operation result according to the destination type of the operation result to obtain standardized data.

[0078] For content that requires subsequent transfer, different standardized data formats are defined based on different destination types. For example, if the delivery destination is an http or https destination, the standardized data format includes the http address, device operation type, tenant information, product information, device information, or data source type. At the same time, the original data collected from the IoT device remains retained and participates in the subsequent data transfer. If encrypted transfer is required, an encrypted transfer identifier and encryption token are added. For example, if the delivery destination is a message queue destination, the standardized data format includes the MQ topic, device operation type, tenant information, product information, device information, or data source type. At the same time, the original data collected from the IoT device remains retained and participates in the subsequent data transfer.

[0079] It should be noted that the destination type can be set by the business process or the user. Specified destinations include but are not limited to HTTP, Pulsar, Kafka, and other destination types.

[0080] Before the operation results are subsequently circulated, the data format is standardized to ensure the integrity and availability of the data.

[0081] S500: Deliver the standardized data to the designated destination and obtain the delivery result.

[0082] In addition to continuously marking the traceId, new delivery destinations and destination types are added to clarify the flow direction. After delivery, callback functions are used to collect and return delivery results. Delivery results include success, failure, timeout, etc.

[0083] Optionally, referring to FIG6 , the lightweight data transfer method of the Internet of Things platform further includes:

[0084] S510, writing the delivery result into the delivery result topic of the message queue for storage;

[0085] and / or,

[0086] S520: Add the delivery result to the standardized data.

[0087] The delivery results are also maintained in a dedicated delivery result topic. While maintaining the traceId, this topic records the delivery result, delivery time, delivery completion time, destination address, tenant information, and device operation type. If the callback result is a failure or timeout, a new failure reason field is added for troubleshooting, data statistics, and analysis. Furthermore, if the result is a failure or timeout, this method provides an alarm function that can refine the problem based on the alarm condition and alarm level.

[0088] It should be noted that step S510 and step S520 can be executed simultaneously, or only one of the steps can be executed.

[0089] The methods in the embodiments of the present invention can be used in series according to business scenario requirements to cover the complete data path.

[0090] In a specific embodiment, referring to FIG7 , the specific process of the lightweight data transfer method of the Internet of Things platform is as follows:

[0091] S1. Data collection: The acquired IoT device data is formed into topics of different formats according to IoT device data, IoT device protocols, and operation types.

[0092] S2. Data Preprocessing: Add the MQ source topic and traceId to the raw data collected by IoT devices. Extract key fields such as tenant information, device information, product information, and operation type from the raw data collected by IoT devices and append these key fields to the raw data collected by IoT devices. The collected raw data is retained and included in subsequent data transfers.

[0093] S3. Data Processing and Analysis: Standardize the results of business process operations based on different destination types. For example, if the destination type is http or https, add the device operation type, tenant information, product information, device information, data source type, etc. to the operation results. If encrypted transfer is required, add an encrypted transfer identifier and encryption token. If the destination type is a message queue, the standardized data format includes the original data of the operation result, MQ topic, device operation type, tenant information, product information, device information, data source type, etc.

[0094] S4. Data delivery preprocessing: Based on the continuous marking of traceId, the delivery destination and destination type are added to clarify the flow direction. After receiving the delivery result, the delivery result is recorded in the data structure.

[0095] S5. Data delivery: Deliver the operation result to the destination according to the destination information based on the data containing the operation result.

[0096] S6. Result callback: After delivery, the delivery result will be returned through the callback function.

[0097] The implementation of the embodiment of the present invention includes the following beneficial effects: this embodiment obtains IoT device data and writes the IoT device data into the topic corresponding to the message queue according to the data format for storage; pre-processes the IoT device data in different topics to obtain pre-processed IoT device data; determines the operation result according to the preset business processing flow and the pre-processed IoT device data; standardizes the operation result according to the destination type of the operation result to obtain standardized data; delivers the standardized data to the specified destination end and obtains the delivery result; converts the IoT device data into a data stream according to the data format for storage. During the data flow process, different data formats are constructed according to the processing purpose to improve data processing efficiency. The business processing flow supports dynamic adjustment of resources to adapt to the real-time requirements of streaming data processing, make up for resource shortages, reduce costs and increase efficiency, and achieve timeliness, integrity and reliability of IoT data transmission; in addition, streaming data processing can be compatible with IoT platform scenarios of different data scales. While having efficient data flow processing capabilities, it streamlines hardware resources, saves memory consumption, and is better compatible with small-scale IoT scenarios, thereby reducing hardware equipment and maintenance costs and improving stability.

[0098] Referring to FIG8 , an embodiment of the present invention provides a lightweight data transfer system for an Internet of Things platform, including:

[0099] The first module is used to obtain IoT device data and write the IoT device data into the topic corresponding to the message queue for storage according to the data format;

[0100] The second module is used to preprocess the IoT device data in different topics to obtain preprocessed IoT device data;

[0101] The third module is used to determine the operation results based on the preset business processing flow and the pre-processed IoT device data;

[0102] The fourth module is used to standardize the operation results according to the destination type of the operation results to obtain standardized data;

[0103] The fifth module is used to deliver the standardized data to the designated destination and obtain the delivery results.

[0104] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0105] Referring to FIG9 , an embodiment of the present invention provides a lightweight data transfer device for an Internet of Things platform, including:

[0106] at least one processor;

[0107] at least one memory for storing at least one program;

[0108] When at least one program is executed by at least one processor, the at least one processor implements the lightweight data transfer method for the Internet of Things platform as claimed in claim 1.

[0109] Among them, the memory is a non-transient computer-readable storage medium that can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a remote memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0110] It can be seen that the contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] In addition, the embodiments of the present application further disclose a computer program product or computer program, which is stored in a computer-readable storage medium. The processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned method. Similarly, the contents of the above-mentioned method embodiment are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above-mentioned method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiment.

[0112] An embodiment of the present invention further provides a computer-readable storage medium, which stores a program executable by a processor. The program executable by the processor is used to implement the above method when executed by the processor.

[0113] It is understood that all or some steps, systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those of ordinary skill in the art, the term computer storage medium is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data) and is volatile and non-volatile, removable and non-removable media. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or can be used to store desired information and any other medium that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0114] 1 , an embodiment of the present invention provides a lightweight data transfer system for an Internet of Things platform, including an Internet of Things device, an Internet of Things platform, and a destination end, wherein the Internet of Things platform connects the Internet of Things device and the destination end;

[0115] IoT devices, used to collect IoT device data and send the IoT device data to the IoT platform;

[0116] An Internet of Things platform, configured to execute the aforementioned lightweight data transfer method for the Internet of Things platform;

[0117] The destination end is used to receive standardized data delivered by the IoT platform and provide feedback on the delivery results.

[0118] Specifically, IoT devices collect IoT device data and send it to the IoT platform in a certain data format; the IoT platform stores, pre-processes, processes business scenarios, processes operation results, and delivers data to IoT device data; the destination end receives the operation results with a standardized structure and feeds back the delivery results to the IoT platform.

[0119] Optionally, the Internet of Things platform includes a data processing unit manager and several data processing units, one data processing unit corresponds to a topic in the message queue, and one data processing unit corresponds to several destination ends.

[0120] Referring to Figure 10, an IoT device operation corresponds to a topic in a message queue. A data processing unit processes a topic in a data source's message queue. The data in the topic is managed by the data processing unit. The data processing unit manager manages multiple data processing units. A data processing unit contains multiple data operators, which perform operations such as transformation, filtering, and aggregation on the collected data. The results are then passed to the next data operator. After processing by multiple data operators, the data is output to a data destination, such as a message queue or HTTP address, or to the next task for further processing. Each data processing unit may interact with multiple delivery destinations.

[0121] Pre-set the parallelism parameters for data reception and data output according to business scenarios, use Nacos dynamic thread pool to quickly configure resources and manage traffic, dynamically adjust the parallelism of data processing operators in real time, adapt to task load fluctuations, reduce task backlogs and delays, ensure that the system performance is always at a high level, and guarantee timely response and processing speed of tasks.

[0122] Data processing units are centrally managed by the Data Processing Unit Manager. This manages the number of active data processing units and controls whether to run or shut down a data processing unit based on the current task execution status, thus avoiding resource waste. Multiple data processing units can process data in parallel within the same task, improving task processing capabilities.

[0123] The method in the embodiment of the present invention can dynamically adjust resources according to the data load to be compatible with data flow management scenarios under different data scales. The method in the embodiment of the present invention is composed of multiple data processing units, each of which can carry one or more parallel tasks.

[0124] The method in the embodiment of the present invention supports various message flow services in the field of Internet of Things, including but not limited to: data dispatch, data push, data forwarding, etc.

[0125] The embodiments of the present invention have the following effects:

[0126] 1. Strive to achieve efficient data flow while minimizing computing and storage resource consumption. Compared with open source frameworks, it is simple and efficient, supports large-scale data flow management needs, and has the conditions for deployment on small IoT platforms, providing high-performance, high-reliability, and low-energy data flow management.

[0127] 2. Propose a standardized data processing format for IoT data flow. Based on the uniqueness of IoT platform data, different data formats are constructed according to the processing purpose in each step of IoT platform data flow. The data retains personalized data such as device protocols, extracts key data such as device information and device operations, and adds important fields such as traceId. While accommodating the differentiation of IoT data, a standardized IoT data format is formed to facilitate subsequent operations.

[0128] 3. Dynamic resource adjustment: Resources can be adjusted dynamically and timely based on resource usage, ensuring that high-occupancy tasks use more resources, while low-occupancy tasks can release excess resources to other tasks, improving resource utilization and thereby increasing data transmission speed and processing capacity. Real-time monitoring is also provided, allowing users to understand the current task processing status based on the monitoring situation and make more detailed adjustments.

[0129] 4. It is more flexible and supports a variety of different types of data sources and transmission destination configuration architectures. It can be integrated with other systems and supports custom data processing rules and processes through the provided business logic processing interface. It has good flexibility and scalability.

[0130] 5. Data transfer tasks in different data formats within the IoT platform are isolated from each other, allowing for parallel processing of multiple data transfer tasks. Each task is independent and does not interfere with each other. Isolating data transfer tasks facilitates standardized processing and improves efficiency while ensuring data security and privacy. The failure or blockage of one task does not affect the normal execution of other tasks, improving efficiency and fault tolerance.

[0131] 6. Strong scalability. Data flow tasks can be arbitrarily deleted and combined through simple operations according to actual needs. Functional integrity is not affected, which is more adaptable to the requirements of the ever-changing data flow situation in the Internet of Things platform.

[0132] It is understood that all or some steps, systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those of ordinary skill in the art, the term computer storage medium is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data) and is volatile and non-volatile, removable and non-removable media. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or can be used to store desired information and any other medium that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0133] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0136] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A lightweight data transfer method for an Internet of Things platform, characterized in that, Including: Obtain Internet of Things (IoT) device data and write the IoT device data into the corresponding topic of a message queue for storage according to the data format; Preprocess the IoT device data in different topics to obtain preprocessed IoT device data; Determine an operation result according to a preset business processing flow and the preprocessed IoT device data; Perform standardization processing on the operation result according to the destination type of the operation result to obtain standardized data; Deliver the standardized data to a specified destination end and obtain a delivery result.

2. The method according to claim 1, wherein The preprocessing includes data format standardization conversion and / or keyword field extraction. The preprocessing of the IoT device data in different topics specifically includes: Add source information to the IoT device data in different topics, where the source information includes a source topic and an information identifier; And / or Extract keyword fields from the IoT device data in different topics and add the keyword fields to the IoT device data.

3. The method according to claim 1, wherein The method further includes: Obtain the business processing flow of a specific business scenario through a business logic processing interface.

4. The method according to claim 1, wherein The business processing flow includes several processing points. The method further includes: When the business processing flow triggers a preset key processing point, write the operation result of the key processing point into the key operation processing information topic of the message queue for storage.

5. The method according to claim 1, wherein The method further includes: Write the delivery result into the delivery result topic of the message queue for storage; And / or Add the delivery result to the standardized data.

6. A lightweight data transfer system for the Internet of Things platform, characterized in that, Including: A first module for obtaining IoT device data and writing the IoT device data into the corresponding topic of a message queue for storage according to the data format; A second module for preprocessing the IoT device data in different topics to obtain preprocessed IoT device data; A third module for determining an operation result according to a preset business processing flow and the preprocessed IoT device data; A fourth module for performing standardization processing on the operation result according to the destination type of the operation result to obtain standardized data; A fifth module for delivering the standardized data to a specified destination and obtaining a delivery result.

7. A lightweight data transfer device for an Internet of Things platform, characterized in that, Including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1-5.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that, The program executable by the processor, when executed by the processor, is used to execute the method according to any one of claims 1-5.

9. A lightweight data transfer system for the Internet of Things platform, characterized in that, Including an IoT device, an IoT platform, and a destination end. The IoT platform connects the IoT device and the destination end; wherein, The IoT device is used to collect IoT device data and send the IoT device data to the IoT platform; The IoT platform is used to execute the method according to any one of claims 1-5; The destination end is used to receive the standardized data delivered by the IoT platform and feedback a delivery result.

10. The system according to claim 9, wherein The Internet of Things platform includes a data processing unit manager and a number of data processing units. One data processing unit corresponds to one topic in the message queue, and one data processing unit corresponds to a number of the destination ends.

Images