Data preprocessing and forwarding system and method based on data flow engine

Abstract

The invention relates to a data preprocessing and forwarding system and method based on a data flow engine, and the system comprises a data preprocessing module and a data forwarding module, the data preprocessing module is responsible for receiving original data from Internet of Things equipment, and carrying out a series of preprocessing operations on the original data; comprising data analysis, format conversion, script processing, data mapping, assembling, filtering and caching, and the data forwarding module is responsible for efficiently forwarding the preprocessed data to various destinations. According to the method, the data preprocessing and forwarding process is highly optimized, and when large-scale Internet of Things data is processed, high efficiency is guaranteed, and the safety and reliability of the data are also guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data preprocessing and forwarding system and method based on a data flow engine. Background Art

[0002] With the rapid development of the Internet of Things (IoT) and big data technologies, we have entered a data-driven era. A large amount of data collected from countless IoT devices and sensors needs to be effectively processed and analyzed to support many applications such as smart cities, industrial automation, and environmental monitoring. These data are not only huge in volume, but also in various formats, including but not limited to JSON, XML, HEX, and ordinary strings. Effectively processing these data and converting them into useful information is crucial to improving operational efficiency, decision support, and user experience.

[0003] However, the existing data processing system has the following problems: (1) The first is the issue of flexibility. The diversity of IoT devices and application scenarios requires that data processing systems be able to adapt to various data formats and processing flows. However, many existing systems lack the necessary flexibility in their design and cannot effectively adapt to these diverse needs.

[0004] (2) The second issue is efficiency. With the rapid growth of data volume, it becomes increasingly difficult to process data quickly and accurately. Many systems experience performance degradation when processing high data traffic and are unable to provide the required data processing services in real time.

[0005] (3) Then there is the issue of adaptability. Existing systems often find it difficult to automatically adapt to ever-changing data characteristics and application requirements, and human intervention is required to adjust processing procedures and algorithms.

[0006] (4) Finally, there is the issue of security. With the increasing number of smart devices and applications, data security and privacy have become an important consideration. Data processing systems need to ensure that data security and user privacy are properly protected during the process of collecting, transmitting and processing data.

[0007] Therefore, there is an urgent need for a data processing system that is efficient, flexible, adaptable and secure. Summary of the invention

[0008] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a data preprocessing and forwarding system and method based on a data flow engine, which can not only flexibly process various data formats, but also efficiently cope with large data traffic, and at the same time have good adaptability and security, support complex data preprocessing, conversion and forwarding tasks, and be suitable for a variety of Internet of Things application scenarios to meet the needs of modern data-intensive applications.

[0009] The object of the present invention is achieved in that: A data preprocessing and forwarding system based on a data flow engine includes a data preprocessing module and a data destination module. The data preprocessing module includes a data receiving unit, a data parsing unit, a script processing unit, a mapping assembly unit, a filtering cache unit and an external data source integration unit. The data destination module includes a data preparation unit, a data destination selection unit, a data forwarding unit and a confirmation log recording unit. The data preprocessing module is used to receive raw data from IoT devices and sensors, and parse, script process, map, assemble, filter, cache and integrate with external data sources. The data destination module is used to forward the preprocessed data to multiple destinations, including but not limited to HTTP services, Redis, Mysql, MongoDB and MQTT servers. Each destination is optimized to handle a specific type of data load. The data destination module has the ability to generate dynamic URLs, supports headers and parameter configuration, and provides access authorization and data encryption functions.

[0010] Furthermore, the data parsing unit adopts a parser for parsing raw data in different formats, including the application of JSONPath, XPath and regular expressions, and supports parsing data formats including HEX, XML and character strings.

[0011] Furthermore, the script processing unit adopts a script processor for running JavaScript scripts.

[0012] Furthermore, the Http service in the data destination module supports dynamic URL generation, headers and parameter configuration, and provides access authorization and data encryption functions.

[0013] Furthermore, in the data destination module, Redis, Mysql or MongoDB is a storage option, which provides a direct storage solution for data and supports dynamic data structure configuration.

[0014] Furthermore, the MQTT server in the data destination module uses message queue technology to provide data consumption services for applications, effectively reducing the concurrency pressure of the system and improving the response efficiency of data processing.

[0015] Based on the above-mentioned data preprocessing and forwarding system based on the data flow engine, the preprocessing and forwarding method includes a data preprocessing method and a data forwarding method. The data preprocessing method includes the following contents: S1, receiving raw data from IoT devices and sensors; S2, preprocessing the raw data using a data preprocessing module; S3, forwarding the pre-processed data to a plurality of set data destinations; Data forwarding methods include the following: S1. Data preparation after preprocessing: After preprocessing, the data is structured into a format suitable for different destinations; S2. Select the data destination: Depending on the nature and purpose of the data, the system will select one or more appropriate destinations; S3, data forwarding: The system transmits data securely to the selected destination by means of dynamic URL generation, configuration of headers and parameters; for transmissions that require high security, the system will implement encryption measures; S4. Confirmation and logging: After the data is forwarded, the system will confirm that the data has been successfully received and record relevant logs for subsequent monitoring and analysis.

[0016] Furthermore, the data preprocessing in step S2 of the data preprocessing method includes the following steps: S2.1. Data analysis: The preprocessing module starts to analyze the data through the parser. For JSON format data, JSONPath is used for parsing; for XML format data, XPath is used for parsing; for HEX and string data, regular expressions are applied; S2.2, Script processing: Use JavaScript scripts to process data; S2.3, Mapping and Assembly: Map specific values ​​in a dataset to a new format or representation, assembling new attribute values ​​as needed; S2.4, Filtering and Cache: Invalid or erroneous data can be filtered out through predefined filtering conditions; at the same time, the caching mechanism is used to optimize data processing efficiency.

[0017] Furthermore, the data preprocessing in step S2 of the data preprocessing method also includes the integration of external data sources: if there is an external data source, the system will integrate data from other applications or systems to enrich and improve the data set.

[0018] Furthermore, in step S2 of the data forwarding method, the destination includes an HTTP service, Redis, Mysql, MongoDB or an MQTT server.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a data preprocessing and forwarding system and method based on a data flow engine, and provides a system including a data flow engine, which is responsible for receiving middle-end data, preprocessing, and converting the processed data into a format suitable for various application systems. The present invention overcomes the problem of low efficiency and accuracy of the IoT data middle-end in the prior art when processing multiple data formats and communication protocols.

[0020] The system design of the present invention enables it to process multiple data formats, adapt to various data processing needs, and provide users with a wide range of customization options. The high flexibility of the system makes it suitable for various complex and changeable data environments; the system design of the present invention has high efficiency through an efficient preprocessing engine and diversified data destination configuration, which not only improves the efficiency of data processing, but also optimizes the data transmission and storage process.

[0021] By integrating advanced data preprocessing technology and flexible data forwarding mechanism, the present invention can quickly process complex data from various IoT devices and effectively support the processing requirements of high concurrency and large data volumes; the present invention not only improves the real-time and reliability of data, but also optimizes the availability and accessibility of data, providing strong data support for intelligent decision-making and automated control.

[0022] The flexibility and high configurability of the present invention enable it to easily adapt to changing technical requirements and business scenarios, ensuring seamless communication between IoT devices and platforms; not only bringing significant economic benefits, but also promoting social benefits such as improving public safety and environmental quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0024] In order to better understand the technical solution of the present invention, the following will be described in detail in conjunction with the relevant illustrations. It should be understood that the following specific embodiments are not intended to limit the specific implementation of the technical solution of the present invention, and are only implementations that can be adopted by the technical solution of the present invention. It should be noted that the description of the positional relationship of each component in this article, such as component A being located above component B, is based on the description of the relative positions of the components in the illustration, and is not intended to limit the actual positional relationship of the components. Example 1

[0025] See also Figure 1 , Figure 1A schematic diagram of the system structure of the present invention is drawn. As shown in the figure, the present invention relates to a data preprocessing and forwarding system based on a data flow engine, including a data preprocessing module and a data destination module (i.e., a data forwarding module), the data preprocessing module includes a data receiving unit, a data parsing unit, a script processing unit, a mapping assembly unit, a filtering cache unit, and an external data source integration unit, the data destination module includes a data preparation unit, a data destination selection unit, a data forwarding unit, and a confirmation log recording unit; The data preprocessing module is used to receive raw data from IoT devices and sensors, and parse, script, map, assemble, filter, cache and integrate external data sources. The data destination module is used to forward the preprocessed data to multiple destinations, including but not limited to HTTP services, Redis, Mysql, MongoDB and MQTT servers. Each destination is optimized to handle a specific type of data load to ensure fast, secure and effective data transmission.

[0026] The data receiving unit may adopt a data receiving server; the data parsing unit may adopt a parser for parsing raw data in different formats, including the application of JSONPath, XPath and regular expressions; the script processing unit may adopt a script processor for running JavaScript scripts to process complex calculation logic and data processing rules; the mapping assembly unit may adopt existing data mapping tools, such as Talend Open Studio, Pentaho Data Integration and IBM InfoSphere; the filtering cache unit may adopt a data filter, such as a Bloom filter; the external data source integration unit may adopt an existing data integration engine, such as the Kettle tool.

[0027] The data destination module has the ability to generate dynamic URLs, supports headers and parameter configuration, and provides access authorization and data encryption functions. The data destination module ensures flexible forwarding and efficient use of data by providing a variety of data destination options.

[0028] The destinations of the data destination module include but are not limited to http services, Redis, Mysql, MongoDB and MQTT servers, and each destination is optimized to handle a specific type of data load to ensure fast, secure and effective data transmission.

[0029] Http service: supports dynamic URL generation, headers and parameter configuration, and provides access authorization and data encryption functions. This function is particularly suitable for scenarios that require high security and customized transmission.

[0030] Storage: You can choose storage options such as Redis, MySQL, and MongoDB to provide direct storage solutions for data and support dynamic data structure configuration. These destinations are suitable for applications that require large-scale data storage and fast access.

[0031] MQTT server: Through message queue technology, the server provides data consumption services for applications, effectively reducing the concurrent pressure of the system and improving the response efficiency of data processing.

[0032] The data preprocessing module first preprocesses the received data. This process covers key steps such as parsing, script processing, mapping, assembly, filtering, caching, and integration of external data sources. The purpose of this preprocessing stage is to convert the raw data into a more structured and standardized format for subsequent processing and analysis. The preprocessed data can be forwarded to a variety of destinations to meet the needs of different application scenarios.

[0033] The present invention relates to a data preprocessing and forwarding method of a data preprocessing and forwarding system based on a data flow engine, including a data preprocessing method and a data forwarding method.

[0034] (1) Data preprocessing methods include the following: S1. Data reception: First, receive the raw data from the IoT device, which can be in multiple formats such as JSON, XML, HEX or plain text strings; Initialize the result set after receiving the data: initialize an empty JSON document as the result set of data processing; for the original message in JSON format, its attributes will be directly filled into this result set, thus providing a basis for subsequent data processing; S2. Data analysis: The preprocessing module begins to analyze data through the parser, supporting the parsing of original messages in various formats such as HEX, XML, and strings. It uses a variety of technical means such as JSONPath, XPath, and regular expressions to flexibly extract necessary information from data in different formats; For data in JSON format, use JSONPath to parse it; for data in XML format, use XPath to parse it; for HEX and string data, apply regular expressions; S3, script processing: Use JavaScript scripts to process data, supporting simple mathematical expressions and more complex functions. This step can include performing mathematical calculations, running custom logic, and other operations, allowing users to customize complex data processing rules and logic, thereby improving the flexibility and accuracy of data processing. S4, mapping and assembly: Mapping portions of the data to new formats or representations, assembling new attribute values ​​as needed; The mapping function allows users to convert specific values ​​in a data set into a new format or representation, while the assembly function supports splicing new attribute values ​​based on custom templates, thereby enhancing the expressiveness and usability of data; S5, filtering and caching: Filter out invalid or erroneous data through predefined filtering conditions; at the same time, use caching mechanisms (such as queues and stacks) to optimize data processing efficiency; The filtering mechanism can effectively determine the validity of messages through user-defined JavaScript expressions or functions, thereby filtering out invalid or erroneous data; the cache mechanism supports two data structures, queue and stack, for efficient management of attributes in a data set or the entire data set. This mechanism is particularly suitable for data scenarios that require frequent access or updates; S6. External data source integration: If there is an external data source, the system will integrate data from other applications or systems to enrich and improve the data set; injecting data from other application software into the data set will make the data richer and more comprehensive, enhancing the value and applicability of the final data.

[0035] (2) Data forwarding methods include the following: S1. Data preparation after preprocessing: After preprocessing, the data is structured into a format suitable for different destinations; S2. Select the data destination: Depending on the nature and purpose of the data, the system will select one or more appropriate destinations, such as HTTP service, Redis, Mysql, MongoDB or MQTT server; S3, data forwarding: The system transmits data securely to the selected destination through dynamic URL generation, configuration of headers and parameters, etc. For transmissions that require high security, the system will implement encryption measures; S4. Confirmation and logging: After the data is forwarded, the system will confirm that the data has been successfully received and record relevant logs for subsequent monitoring and analysis.

[0036] The present invention relates to a data preprocessing and forwarding system based on a data flow engine, which can be applied to an Internet of Things data middle-office system, and provides a comprehensive and effective data middle-office solution in the field of Internet of Things technology. It is particularly suitable for various application scenarios such as smart cities, industrial automation, and environmental monitoring, and can handle large amounts of data and high-concurrency data processing requirements. The core of this solution is to significantly improve the efficiency and accuracy of data processing, while also greatly simplifying the integration process with downstream systems.

[0037] Working principle: The present invention provides a data preprocessing and forwarding system and method based on a data flow engine, which is specially designed for the needs of the Internet of Things data center. The core is an innovative data flow engine that aims to improve the processing efficiency and accuracy of data collected by Internet of Things devices and sensors.

[0038] The system of the present invention includes two main parts: a data pre-processing module and a data forwarding module (ie, a data destination module).

[0039] The data preprocessing module is responsible for receiving the raw data from IoT devices and performing a series of preprocessing operations on it, including data parsing, format conversion, script processing, data mapping, assembly, filtering and caching. In addition, this module also integrates external data sources to enrich the data content and improve the data structure. These processing steps ensure that the data is effectively standardized and optimized before forwarding, providing a reliable foundation for subsequent analysis and application.

[0040] The data forwarding module is responsible for efficiently forwarding the pre-processed data to various destinations, such as HTTP services, databases (such as Redis, Mysql, MongoDB) and MQTT servers, etc. This module ensures secure data transmission and flexible application through functions such as dynamic URL generation, parameter configuration and data encryption.

[0041] The core of the system of the present invention is the high optimization of data preprocessing and forwarding process. The system can adapt to various data formats, support complex data processing logic, and can flexibly adapt to different data destination requirements. These features enable the system to ensure high efficiency, data security and reliability when processing large-scale IoT data.

[0042] The system of the present invention is particularly suitable for data-intensive Internet of Things applications, such as smart city construction, industrial automation, and environmental monitoring. It can effectively support the high-concurrency data processing requirements in these applications, provide solid data support for intelligent decision-making and automated control, and simplify the data processing and forwarding processes, thereby improving the operating efficiency of the overall system.

[0043] The above are only specific application examples of the present invention and do not constitute any limitation on the protection scope of the present invention. Any technical solution formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention.

Claims

1. A data preprocessing and forwarding system based on a data flow engine, characterized in that: It includes a data preprocessing module and a data destination module. The data preprocessing module includes a data receiving unit, a data parsing unit, a script processing unit, a mapping assembly unit, a filtering cache unit and an external data source integration unit. The data destination module includes a data preparation unit, a data destination selection unit, a data forwarding unit and a confirmation log recording unit. The data preprocessing module is used to receive the original data of the Internet of Things devices and sensors, and parse, script process, map, assemble, filter, cache and integrate the data with the external data source. The data destination module is used to forward the preprocessed data to multiple destinations, including but not limited to HTTP services, Redis, Mysql, MongoDB and MQTT servers. Each destination is optimized to handle a specific type of data load. The data destination module has the ability to generate dynamic URLs, supports headers and parameter configuration, and provides access authorization and data encryption functions.

2. According to claim 1, a data preprocessing and forwarding system based on a data flow engine is characterized in that: The data parsing unit adopts a parser for parsing raw data in different formats, including the application of JSONPath, XPath and regular expressions, and supports parsing data formats including HEX, XML and character strings.

3. The data preprocessing and forwarding system based on the data flow engine according to claim 1 is characterized in that: The script processing unit adopts a script processor for running JavaScript scripts.

4. The data preprocessing and forwarding system and method based on the data flow engine according to claim 1, characterized in that: The Http service in the data destination module supports dynamic URL generation, headers and parameter configuration, and provides access authorization and data encryption functions.

5. The data preprocessing and forwarding system and method based on the data flow engine according to claim 1, characterized in that: In the data destination module, Redis, Mysql or MongoDB is a storage option, which provides a direct storage solution for data and supports dynamic data structure configuration.

6. The data preprocessing and forwarding system and method based on the data flow engine according to claim 1, characterized in that: The MQTT server in the data destination module uses message queue technology to provide data consumption services for applications, effectively reducing the concurrency pressure of the system and improving the response efficiency of data processing.

7. A preprocessing and forwarding method of a data preprocessing and forwarding system based on a data flow engine according to claim 1, characterized in that: It includes a data preprocessing method and a data forwarding method. The data preprocessing method includes the following contents: S1, receiving raw data from IoT devices and sensors; S2, preprocessing the raw data using a data preprocessing module; S3, forwarding the pre-processed data to a plurality of set data destinations; Data forwarding methods include the following: S1. Data preparation after preprocessing: After preprocessing, the data is structured into a format suitable for different destinations; S2. Select the data destination: Depending on the nature and purpose of the data, the system will select one or more appropriate destinations; S3, data forwarding: The system transmits data securely to the selected destination by means of dynamic URL generation, configuration of headers and parameters; for transmissions that require high security, the system will implement encryption measures; S4. Confirmation and logging: After the data is forwarded, the system will confirm that the data has been successfully received and record relevant logs for subsequent monitoring and analysis.

8. The preprocessing and forwarding method of the data preprocessing and forwarding system based on the data flow engine according to claim 7, characterized in that: The data preprocessing in step S2 of the data preprocessing method includes the following steps: S2.

1. Data analysis: The preprocessing module starts to analyze the data through the parser. For JSON format data, JSONPath is used for parsing; for XML format data, XPath is used for parsing; for HEX and string data, regular expressions are applied; S2.2, Script processing: Use JavaScript scripts to process data; S2.3, Mapping and Assembly: Map specific values ​​in a dataset to a new format or representation, assembling new attribute values ​​as needed; S2.4, Filtering and Cache: Invalid or erroneous data can be filtered out through predefined filtering conditions; at the same time, the caching mechanism is used to optimize data processing efficiency.

9. The preprocessing and forwarding method of the data preprocessing and forwarding system based on the data flow engine according to claim 7, characterized in that: In step S2 of the data preprocessing method, the data preprocessing also includes the integration of external data sources: if there is an external data source, the system will integrate data from other applications or systems to enrich and improve the data set.

10. The preprocessing and forwarding method of the data preprocessing and forwarding system based on the data flow engine according to claim 7, characterized in that: In step S2 of the data forwarding method, the destination includes an HTTP service, Redis, Mysql, MongoDB or an MQTT server.

Images