Cross-platform multi-source heterogeneous data fusion system and method

Abstract

The invention relates to the technical field of heterogeneous data processing, in particular to a cross-platform multi-source heterogeneous data fusion system and method.The system comprises a server side, a user side and data acquisition sides; the server side comprises a generation module used for acquiring configuration information uploaded by the user side and generating corresponding heterogeneous data acquisition sub-modules based on the configuration information; the heterogeneous data acquisition sub-module is used for sending a data acquisition signal to a corresponding data acquisition end, acquiring heterogeneous data corresponding to the corresponding data acquisition end, configuring a corresponding protocol key, performing data packaging on the heterogeneous data and the corresponding protocol key, forming a corresponding data packet, and feeding back the data packet to the server; and the data analysis module is used for identifying the protocol key in the data packet according to the data packet, calling an analysis protocol corresponding to the protocol key from the protocol database, and analyzing the heterogeneous data in the data packet through the analysis protocol to form corresponding analysis data.

Description

Technical Field

[0001] The present invention relates to the technical field of heterogeneous data processing, and in particular to a cross-platform multi-source heterogeneous data fusion system and method. Background Art

[0002] With the rapid development of information technology, various sensors, IoT devices and other data acquisition devices are widely used in industrial production, urban management, environmental protection and other fields. However, these devices often generate a large amount of heterogeneous data, that is, data from different sources and in different formats. How to effectively collect, process and fuse these heterogeneous data has become an urgent problem to be solved.

[0003] Traditional data fusion processing systems can usually only process data in a specific format. When faced with data from multiple different sources, additional data conversion and adaptation work is often required, which not only increases the complexity of the system, but also limits the real-time processing capabilities and application scope of the data. In addition, due to the lack of a unified data management and feedback mechanism, there may be delays, losses or errors in the data processing process, affecting the accuracy and timeliness of the data. Summary of the invention

[0004] One of the purposes of the present invention is to provide a cross-platform multi-source heterogeneous data fusion system and method, which can realize unified data management of cross-platform multi-source heterogeneous data and improve the accuracy and efficiency of the data fusion process.

[0005] In order to achieve the above objectives, a cross-platform multi-source heterogeneous data fusion system is provided, including a server, a user, and various data collection terminals;

[0006] The server includes:

[0007] A generation module is used to obtain the configuration information uploaded by the user end, and based on the configuration information, generate corresponding heterogeneous data acquisition submodules; the heterogeneous data acquisition submodule is used to send a data acquisition signal to the corresponding data acquisition end, and based on the data acquisition signal, collect the heterogeneous data corresponding to the corresponding data acquisition end and configure the corresponding protocol key, and package the heterogeneous data and the corresponding protocol key to form a corresponding data packet and feed it back to the server end at the same time;

[0008] The data parsing module is used to identify the protocol key in the data packet according to the formed data packet, and based on the identified protocol key, retrieve the parsing protocol corresponding to the protocol key from the protocol database, parse the heterogeneous data in the data packet through the parsing protocol, and form parsed data corresponding to the corresponding heterogeneous data;

[0009] The feedback module is used to feed back the parsed data corresponding to the corresponding heterogeneous data to the user end and the corresponding data collection end.

[0010] Technical principles and effects of this solution: In this solution, first, the configuration information uploaded by the user end is obtained through the generation module, and each heterogeneous data acquisition sub-module is determined through the configuration information, so as to realize the definition of the data acquisition interface between different data, and then the heterogeneous data acquisition sub-module is used to send data acquisition signals to the corresponding data acquisition ends. At this time, after receiving the data acquisition signal, the data acquisition end will send the corresponding heterogeneous data to the heterogeneous data acquisition sub-module in the server end and configure the protocol key corresponding to each heterogeneous data, and feedback the corresponding data packet to the server end by packaging the heterogeneous data and the protocol key.

[0011] The data parsing module will then identify the protocol key in the data packet, retrieve the parsing protocol corresponding to the protocol key from the protocol database through the identified protocol key, and then parse the heterogeneous data in the data packet through the parsing protocol, so as to realize the parsing of heterogeneous data, form corresponding parsing data and feed it back to the user end and the corresponding data collection end.

[0012] The generation module can automatically create heterogeneous data acquisition submodules suitable for different data acquisition terminals based on the configuration information uploaded by the user terminal, reducing the workload of manual configuration. By sending data acquisition signals to the corresponding data acquisition terminals, the system can flexibly control the time and frequency of data acquisition to meet the needs of different scenarios.

[0013] During the data collection process, a corresponding protocol key is configured for each piece of heterogeneous data to ensure the unique identification of the data. Heterogeneous data and its protocol key are packaged into data packets to facilitate subsequent data analysis and processing, ensuring the consistency and integrity of the data. Through the combined use of protocol keys and analysis protocols, the system can perform integrity checks on the data to ensure the correctness and reliability of the data, achieving unified data management and data conversion for multi-source heterogeneous data across platforms, and improving the accuracy and efficiency of the data fusion process.

[0014] Furthermore, the server also includes a relay module for acquiring data acquisition signals sent by each heterogeneous data acquisition submodule to the corresponding data acquisition terminal and sending the acquired data acquisition signals to the corresponding data acquisition terminal;

[0015] It is also used to obtain the heterogeneous data corresponding to each data acquisition end after each data acquisition end receives the corresponding data acquisition signal, convert the corresponding heterogeneous data into a format that can be received by the server, and transmit the converted heterogeneous data to the corresponding heterogeneous data acquisition sub-module.

[0016] Beneficial effects: In this solution, through the relay module, the system can handle multiple different types of data sources, which means that it can be more easily integrated with the existing infrastructure without the need to adapt each data source separately. Since the relay module is responsible for converting different heterogeneous data into a format recognizable by the server, this eliminates the problem of data inconsistency and simplifies the subsequent data processing process.

[0017] Furthermore, the server also includes a data storage module, which is used to arrange the parsing order corresponding to each data packet based on the data packet parsing arrangement strategy, and form a corresponding data packet parsing order table;

[0018] It is also used to store the data packets in the data packet parsing sequence table according to the formed data packet parsing sequence table; the parsing module is used to retrieve the corresponding formed data packets from the data storage module in sequence according to the data parsing sequence table;

[0019] The data packet parsing and arranging strategy is:

[0020] Within a preset time period, based on a preset data importance calculation formula, the data importance corresponding to each data packet formed within the preset time period is calculated;

[0021] The data importance calculation formula is:

[0022]

[0023] In the formula, Z is the data importance corresponding to the data packet, F is the data freshness corresponding to the data packet, D is the importance level of the data collection end corresponding to the data packet, and T d is the current time corresponding to the processing of the data packet, T S is the generation time corresponding to the data packet generation; w1, w2 are the corresponding weighting coefficients;

[0024] According to the data importance corresponding to each data packet, the data are sorted in order from large to small to form a corresponding data analysis sequence table.

[0025] Beneficial effects: In this solution, the data packets corresponding to the preset time period are sorted according to the importance of the data. In this way, effective processing resources can be reasonably allocated to avoid wasting resources on processing low-priority data. For data that needs to be processed in real time or near real time, this strategy can help reduce processing delays and ensure timely decision-making. The system can ensure that data packets with higher freshness and higher collection end importance are processed first, thereby improving the system's response speed and service quality. If a data packet cannot be processed temporarily because of its low importance, the system can process more important data packets first, so that even if an error occurs, it will not seriously affect the overall performance.

[0026] Further, the feedback module includes:

[0027] A first sending module, used for sending parsed data corresponding to the corresponding heterogeneous data to a user terminal;

[0028] The second sending module is used to identify the reply mode in the configuration information of the heterogeneous data corresponding to the parsed data, and based on the identified reply mode, retrieve the corresponding reply strategy and send the corresponding parsed data to the corresponding data collection end; the reply mode includes a long-period reply mode, a short-period reply mode and a no-reply mode;

[0029] The response strategy is:

[0030] If the reply mode is no reply mode, the corresponding server only receives the data collection end, and the server does not send data back to the corresponding data collection end;

[0031] If the response mode is a short-term response mode, the server will immediately feed back the parsed data to the data collection end;

[0032] If the reply mode is the long-cycle mode, the server will identify the analysis processing logic corresponding to the corresponding analysis data, package the analysis data and the analysis processing logic into a corresponding reply package, and send the reply package to the corresponding data collection end.

[0033] Beneficial effects: In this solution, by distinguishing different response modes (no response mode, short-cycle response mode, and long-cycle response mode), the system can better meet the specific needs of different data acquisition terminals and provide personalized services. For situations where immediate feedback is required, the short-cycle response mode can quickly return the analysis results to the data acquisition terminal, which is very useful for applications that require real-time monitoring. The no-response mode is suitable for situations where feedback confirmation is not required, reducing unnecessary communication overhead and saving network resources. The long-cycle response mode allows the server to perform more complex analysis and processing logic and return the results together with the logic itself to the data acquisition terminal. This is very useful for scenarios that require data analysis, prediction, or other complex calculations, and greatly improves the adaptability of the system.

[0034] The present invention also provides a cross-platform multi-source heterogeneous data fusion method, using the above-mentioned cross-platform multi-source heterogeneous data fusion system, comprising the following steps:

[0035] S1. Obtain the configuration information uploaded by the user end, and generate corresponding heterogeneous data acquisition submodules based on the configuration information; the heterogeneous data acquisition submodule is used to send a data acquisition signal to the corresponding data acquisition end, and based on the data acquisition signal, collect the heterogeneous data corresponding to the corresponding data acquisition end and configure the corresponding protocol key, and package the heterogeneous data and the corresponding protocol key to form a corresponding data packet;

[0036] S2. According to the formed data packet, the protocol key in the data packet is identified, and based on the identified protocol key, the parsing protocol corresponding to the protocol key is retrieved from the protocol database, and the heterogeneous data in the data packet is parsed through the parsing protocol to form parsed data corresponding to the heterogeneous data;

[0037] S3. Feedback the parsed data corresponding to the corresponding heterogeneous data to the user end and the corresponding data collection end. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a logic block diagram of a cross-platform multi-source heterogeneous data fusion system in Embodiment 1 of the present invention;

[0039] Figure 2 This is a flow chart of a cross-platform multi-source heterogeneous data fusion method in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0040] The following is further described in detail through specific implementation methods:

[0041] Embodiment 1

[0042] Cross-platform multi-source heterogeneous data fusion system and method, basically Figure 1As shown, it includes a server, a user and various data collection terminals;

[0043] The server includes:

[0044] A generation module is used to obtain the configuration information uploaded by the user end, and based on the configuration information, generate corresponding heterogeneous data acquisition sub-modules; the heterogeneous data acquisition sub-module is used to send a data acquisition signal to the corresponding data acquisition end, and based on the data acquisition signal, collect the heterogeneous data corresponding to the corresponding data acquisition end and configure the corresponding protocol key, and package the heterogeneous data and the corresponding protocol key to form a corresponding data packet and feed it back to the server end at the same time; in this embodiment, the configuration information includes the corresponding heterogeneous data, and the protocol key, parsing protocol, and reply mode corresponding to the heterogeneous data.

[0045] The server also includes a relay module for acquiring data acquisition signals sent by each heterogeneous data acquisition submodule to the corresponding data acquisition terminal and sending the acquired data acquisition signals to the corresponding data acquisition terminal;

[0046] It is also used to obtain the heterogeneous data corresponding to each data acquisition terminal after each data acquisition terminal receives the corresponding data acquisition signal, and convert the corresponding heterogeneous data into a format that can be received by the server, and transmit the heterogeneous data after the conversion to the corresponding heterogeneous data acquisition submodule. In this embodiment, different data acquisition terminals correspond to different communication methods. For example, the communication method corresponding to the street lamp lighting control terminal, the hazard source monitoring terminal, and the smart park comprehensive monitoring terminal is TCP / IP, while the communication method corresponding to the bridge monitoring terminal, the forest fire prevention terminal, and the geological disaster monitoring terminal is httpserver. Of course, some data acquisition terminals also use Bluetooth communication. Based on this, the unification of different communication methods is achieved through the setting of the relay module, and the heterogeneous data corresponding to the corresponding data acquisition terminal can be accurately transmitted to the server.

[0047] The server also includes a data storage module for arranging the parsing order corresponding to each data packet based on the data packet parsing arrangement strategy to form a corresponding data packet parsing order table;

[0048] It is also used to store the data packets in the data packet parsing sequence table according to the formed data packet parsing sequence table; the parsing module is used to retrieve the corresponding formed data packets from the data storage module in sequence according to the data parsing sequence table;

[0049] The data packet parsing and arranging strategy is:

[0050] Within a preset time period, based on a preset data importance calculation formula, the data importance corresponding to each data packet formed within the preset time period is calculated;

[0051] The data importance calculation formula is:

[0052]

[0053] In the formula, Z is the data importance corresponding to the data packet, F is the data freshness corresponding to the data packet, D is the importance level of the data collection end corresponding to the data packet, and T d is the current time corresponding to the processing of the data packet, T S is the generation time corresponding to the data packet generation; w1, w2 are the corresponding weighting coefficients;

[0054] According to the data importance corresponding to each data packet, the data are sorted in order from large to small to form a corresponding data analysis sequence table.

[0055] The data parsing module is used to identify the protocol key in the data packet according to the formed data packet, and based on the identified protocol key, retrieve the parsing protocol corresponding to the protocol key from the protocol database, parse the heterogeneous data in the data packet through the parsing protocol, and form parsed data corresponding to the corresponding heterogeneous data;

[0056] The feedback module is used to feed back the parsed data corresponding to the corresponding heterogeneous data to the user end and the corresponding data collection end.

[0057] The feedback module comprises:

[0058] A first sending module, used for sending parsed data corresponding to the corresponding heterogeneous data to a user terminal;

[0059] The second sending module is used to identify the reply mode in the configuration information of the heterogeneous data corresponding to the parsed data, and based on the identified reply mode, retrieve the corresponding reply strategy and send the corresponding parsed data to the corresponding data collection end; the reply mode includes a long-period reply mode, a short-period reply mode and a no-reply mode;

[0060] The response strategy is:

[0061] If the reply mode is no reply mode, the corresponding server only receives the data collection end, and the server does not send data back to the corresponding data collection end;

[0062] If the reply mode is a short-cycle reply mode, the server will immediately feed back the parsed data to the data collection end; in this embodiment, the short-cycle reply mode means that if the time period required for the interaction is short and the reply logic is simple, this type of data can be replied to immediately.

[0063] If the reply mode is a long-cycle mode, the server will identify the parsing processing logic corresponding to the corresponding parsing data, package the parsing data and the parsing processing logic to form a corresponding reply package, and send the reply package to the corresponding data collection end. In this embodiment, if the interaction requires a reply, and the reply requires parsing processing logic, the reply package will be cached after parsing at the parsing end, and then obtained by the collection end and sent back to the device. At the same time, the collection management module and the parsing management module are required to have a higher scheduling frequency, and the access interface of various types of interactive data is set with data caching logic to improve performance.

[0064] like Figure 2 As shown, this embodiment also discloses a cross-platform multi-source heterogeneous data fusion method, using the above-mentioned cross-platform multi-source heterogeneous data fusion system, including the following steps:

[0065] S1. Obtain the configuration information uploaded by the user end, and generate corresponding heterogeneous data acquisition submodules based on the configuration information; the heterogeneous data acquisition submodule is used to send a data acquisition signal to the corresponding data acquisition end, and based on the data acquisition signal, collect the heterogeneous data corresponding to the corresponding data acquisition end and configure the corresponding protocol key, and package the heterogeneous data and the corresponding protocol key to form a corresponding data packet;

[0066] S2. According to the formed data packet, the protocol key in the data packet is identified, and based on the identified protocol key, the parsing protocol corresponding to the protocol key is retrieved from the protocol database, and the heterogeneous data in the data packet is parsed through the parsing protocol to form parsed data corresponding to the heterogeneous data;

[0067] S3. Feedback the parsed data corresponding to the corresponding heterogeneous data to the user end and the corresponding data collection end.

[0068] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is described too much here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can know all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A cross-platform multi-source heterogeneous data fusion system, characterized by: Including the server, user and various data collection terminals; The server includes: A generation module is used to obtain the configuration information uploaded by the user end, and based on the configuration information, generate corresponding heterogeneous data acquisition submodules; the heterogeneous data acquisition submodule is used to send a data acquisition signal to the corresponding data acquisition end, and based on the data acquisition signal, collect the heterogeneous data corresponding to the corresponding data acquisition end and configure the corresponding protocol key, and package the heterogeneous data and the corresponding protocol key to form a corresponding data packet and feed it back to the server end at the same time; The data parsing module is used to identify the protocol key in the data packet according to the formed data packet, and based on the identified protocol key, retrieve the parsing protocol corresponding to the protocol key from the protocol database, parse the heterogeneous data in the data packet through the parsing protocol, and form parsed data corresponding to the corresponding heterogeneous data; The feedback module is used to feed back the parsed data corresponding to the corresponding heterogeneous data to the user end and the corresponding data collection end.

2. The cross-platform multi-source heterogeneous data fusion system according to claim 1 is characterized by: The server also includes a relay module for acquiring data acquisition signals sent by each heterogeneous data acquisition submodule to the corresponding data acquisition terminal and sending the acquired data acquisition signals to the corresponding data acquisition terminal; It is also used to obtain the heterogeneous data corresponding to each data acquisition end after each data acquisition end receives the corresponding data acquisition signal, convert the corresponding heterogeneous data into a format that can be received by the server, and transmit the converted heterogeneous data to the corresponding heterogeneous data acquisition sub-module.

3. The cross-platform multi-source heterogeneous data fusion system according to claim 2 is characterized by: The server also includes a data storage module for arranging the parsing order corresponding to each data packet based on the data packet parsing arrangement strategy, so as to form a corresponding data packet parsing order table; It is also used to store the data packets in the data packet parsing sequence table according to the formed data packet parsing sequence table; the parsing module is used to retrieve the corresponding formed data packets from the data storage module in sequence according to the data parsing sequence table; The data packet parsing and arranging strategy is: Within a preset time period, based on a preset data importance calculation formula, the data importance corresponding to each data packet formed within the preset time period is calculated; The data importance calculation formula is: In the formula, Z is the data importance corresponding to the data packet, F is the data freshness corresponding to the data packet, D is the importance level of the data collection end corresponding to the data packet, and T d is the current time corresponding to the processing of the data packet, T S is the generation time corresponding to the data packet generation; w1, w2 are the corresponding weighting coefficients; According to the data importance corresponding to each data packet, the data are sorted in order from large to small to form a corresponding data analysis sequence table.

4. The cross-platform multi-source heterogeneous data fusion system according to claim 3 is characterized by: The feedback module comprises: A first sending module, used for sending parsed data corresponding to the corresponding heterogeneous data to a user terminal; The second sending module is used to identify the reply mode in the configuration information of the heterogeneous data corresponding to the parsed data, and based on the identified reply mode, retrieve the corresponding reply strategy and send the corresponding parsed data to the corresponding data collection end; the reply mode includes a long-period reply mode, a short-period reply mode and a no-reply mode; The response strategy is: If the reply mode is no reply mode, the corresponding server only receives the data collection end, and the server does not send data back to the corresponding data collection end; If the response mode is a short-term response mode, the server will immediately feed back the parsed data to the data collection end; If the reply mode is the long-cycle mode, the server will identify the analysis processing logic corresponding to the corresponding analysis data, package the analysis data and the analysis processing logic into a corresponding reply package, and send the reply package to the corresponding data collection end.

5. A cross-platform multi-source heterogeneous data fusion method, using the cross-platform multi-source heterogeneous data fusion system according to any one of claims 1 to 4, characterized in that: The following steps are involved: S1. Obtain the configuration information uploaded by the user end, and generate corresponding heterogeneous data acquisition submodules based on the configuration information; the heterogeneous data acquisition submodule is used to send a data acquisition signal to the corresponding data acquisition end, and based on the data acquisition signal, collect the heterogeneous data corresponding to the corresponding data acquisition end and configure the corresponding protocol key, and package the heterogeneous data and the corresponding protocol key to form a corresponding data packet; S2. According to the formed data packet, the protocol key in the data packet is identified, and based on the identified protocol key, the parsing protocol corresponding to the protocol key is retrieved from the protocol database, and the heterogeneous data in the data packet is parsed through the parsing protocol to form parsed data corresponding to the heterogeneous data; S3. Feedback the parsed data corresponding to the corresponding heterogeneous data to the user end and the corresponding data collection end.

Images