Multi-source heterogeneous data fusion method and system, electronic equipment and medium

By monitoring data ports in real-time in water conservancy projects, verifying and fusing preset standard attributes in text data, the text data storage problem in multi-source heterogeneous data fusion is solved, the data format uniformity and fusion are realized, and data management is improved efficiency.

CN118940208APending Publication Date: 2024-11-12GUIZHOU SMART WATER CONSERVANCY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410956551.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In water conservancy projects, text types of data from multiple sources cannot be stored in a unified format, resulting in difficulty in fusion of multi-source heterogeneous data and affecting data management and system operation.

Method used

By monitoring the preset ports in real time, receiving data, verifying data type, obtaining preset standard attributes in text as target data, and integrating them with other data types, storing them in the database, unifying text data from different formats and sources.

Benefits of technology

The format of text data is unified, which facilitates the fusion of storage and multi-source heterogeneous data, and solves the problem of heterogeneous data fusion in data management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940208A_ABST
    Figure CN118940208A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of water conservancy monitoring, and discloses a multi-source heterogeneous data fusion method and system, electronic equipment and a medium, and the method comprises the steps: monitoring a preset port in real time; wherein the preset port is used for receiving data sent by each monitoring device; performing data type verification on the currently received data; if the currently received data type comprises the text, acquiring data corresponding to the preset standard attribute in the text, and determining target data; and performing multi-source heterogeneous data fusion on the currently received target data and the data corresponding to the other data types, determining fused data, and storing the fused data in a database. According to the method and the device, the preset standard attribute in the text is taken as a standard, so that the text data with different formats and different sources are subjected to format unification, and data storage and multi-source heterogeneous data fusion are facilitated.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] In water conservancy projects, dynamic monitoring data is very large, and the relationship between calling and extracting data of different types and sources is also very complicated. Since the construction of various business systems and the implementation of data management have certain technical and periodic problems, data will face more or less "business barriers" and application difficulties during system operation, resulting in the accumulation of a large amount of data information using different storage methods. In addition, if the traditional integration method is adopted, the data will be divided in the form of multiple data management systems, which leads to the generation of heterogeneous data sources in the entire project standardized operation management system.

[0003] Especially for text-type data, the data information comes from various channels, including front-end sensors, external input and system operation, which makes it impossible to store text-type data in a unified format, thus affecting the fusion of multi-source heterogeneous data. Summary of the invention

[0004] In order to overcome the problem that existing nondestructive testing technology cannot detect early damage such as stress concentration and lacks a basis for judging displacement and deformation of pipelines caused by stress, the present invention provides a multi-source heterogeneous data fusion method, system, electronic equipment and medium.

[0005] In a first aspect, in order to solve the above technical problems, the present invention provides a multi-source heterogeneous data fusion method, comprising:

[0006] Monitor the preset port in real time; the preset port is used to receive data sent by various monitoring devices;

[0007] Verify the data type of the currently received data;

[0008] If the currently received data type includes text, then the data corresponding to the preset standard attribute in the text is obtained to determine the target data;

[0009] Perform multi-source heterogeneous data fusion on the currently received target data and data corresponding to other data types, determine the fused data, and store the fused data in the database.

[0010] In a second aspect, the present invention provides a multi-source heterogeneous data fusion system, comprising:

[0011] A monitoring module is used to monitor a preset port in real time; wherein the preset port is used to receive data sent by each monitoring device;

[0012] A verification module is used to verify the data type of the currently received data;

[0013] A target data determination module is used to obtain data corresponding to a preset standard attribute in the text to determine the target data if the currently received data type includes text;

[0014] The multi-source heterogeneous data fusion module is used to fuse the currently received target data with the data corresponding to other data types, determine the fused data, and store the fused data in the database.

[0015] In a third aspect, the present invention provides a computing device, including a memory, a processor, and a program stored in the memory and running on the processor, and when the processor executes the program, the steps of a multi-source heterogeneous data fusion method as described above are implemented.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the steps of a multi-source heterogeneous data fusion method as described above.

[0017] The beneficial effect of the present invention is that when data received by a preset port is monitored, the data type is verified. If the data is of text type, data corresponding to the preset standard attribute is obtained from the text, thereby obtaining the target data, and multi-source heterogeneous data fusion is performed with data of other data types and stored in a database. The present application uses the preset standard attribute in the text as a standard, and then unifies the format of text data of different formats and sources, which is convenient for data storage and fusion of multi-source heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0019] Figure 1 A schematic diagram of a multi-source heterogeneous data fusion method according to an embodiment of the present invention;

[0020] Figure 2 The figure is a schematic diagram of the structure of a multi-source heterogeneous data fusion system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following examples are provided to further explain and supplement the present invention and do not constitute any limitation to the present invention.

[0022] A multi-source heterogeneous data fusion method, system, electronic device, and medium according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, the present invention provides a multi-source heterogeneous data fusion method, comprising:

[0024] S1. Monitor the preset port in real time; wherein the preset port is used to receive data sent by various monitoring devices.

[0025] S2. Check the data type of the currently received data.

[0026] S3. If the currently received data type includes text, then the data corresponding to the preset standard attribute in the text is obtained to determine the target data.

[0027] S4. Perform multi-source heterogeneous data fusion on each target data currently received, determine the fused data, and store the fused data in a database.

[0028] In this embodiment, when the data received by the preset port is monitored, the data type is verified. If the data is of text type, the data corresponding to the preset standard attribute is obtained from the text, thereby obtaining the target data, and multi-source heterogeneous data fusion is performed with data of other data types and stored in the database. This application uses the preset standard attribute in the text as a standard, and then unifies the format of text data of different formats and sources, which is convenient for data storage and fusion of multi-source heterogeneous data.

[0029] In this embodiment, the types of monitoring equipment from different manufacturers are different. Even the same type of monitoring may result in different data formats due to different production batches. However, since the mainstream communication protocols include TCP / IP protocol, Modbus protocol, CAN protocol, etc., in this embodiment, the monitoring equipment of each manufacturer is required to transmit data based on the above communication protocols.

[0030] In this embodiment, the preset standard attributes may be conventional parameters in the field of water conservancy projects, such as transmission time, station code, station type, etc., which almost cover the parameters involved in the monitoring equipment of each manufacturer in this field.

[0031] Optionally, obtaining data corresponding to a preset standard attribute in the text and determining target data includes:

[0032] Get the data corresponding to each preset field in the text;

[0033] The acquired data corresponding to each preset field is used as the target data.

[0034] For example, taking the SL 651-2014 protocol (hydrological monitoring data communication protocol) as an example, the text data monitored to the preset port includes:

[0035] 7E 7E 00 00 00 00 00 22 00 00 32 00 2B 02 00 01 24 03 05 15 00 35F1F100 00 00 00 22 48F0F0 19 03 05 15 00 22 19 00 00 50 1A 19 00 01 50 20 1900 0030 26 19 00 07 50 03 8A 75.

[0036] The preset standard attributes include the station code "siteId", the station type "siteType", the observation time "dateTime", the precipitation in one hour "hour", the current precipitation "current" and the cumulative precipitation value "cumulative".

[0037] Based on the above, since each manufacturer transmits data based on the above communication protocol, the preset field will contain data corresponding to the above preset standard attributes, and the data corresponding to the preset standard attributes will be stored in the preset position (preset field) in the text. Therefore, it is only necessary to find the data in these preset fields and extract them to unify the format and obtain the following target data:

[0038] "siteId": "0000000022".

[0039] "siteType": "river".

[0040] "dateTime": "2024-03-05 15:00:35".

[0041] "5-minute precipitation": "5"

[0042] "hour": "15".

[0043] "current": "3".

[0044] "cumulative": "75".

[0045] "unit": "mm".

[0046] It should be noted that if the data corresponding to the preset standard attribute is a number, but the data type of the target data is a letter, data conversion is required. For example, the preset standard attribute represents the station type, which includes rivers, land, and oceans. Then 01 can be used to represent rivers, 02 to represent rainfall stations, and 03 to represent water level station identifiers. When the data corresponding to the preset field is 01, 01 needs to be converted to letters, namely river.

[0047] Optionally, obtaining data corresponding to each preset field in the text also includes:

[0048] If the data corresponding to any preset field in this article is missing, the corresponding missing data is requested from the business platform corresponding to the sent text.

[0049] Although the above communication protocol adopted by various manufacturers in this embodiment transmits data, some preset standard attributes may be missing, for example, the data cannot be obtained due to device damage, or the device does not have this function.

[0050] For example, the preset standard attributes include 5-minute precipitation "5-minute precipitation" and precipitation unit "unit", but the corresponding data cannot be obtained from the preset fields in the text, so it is necessary to request the corresponding missing data from the business platform.

[0051] If the device in the business platform has the above preset standard attributes, the corresponding data will be returned, for example, "5-minute precipitation": "5", precipitation "unit": "mm".

[0052] If the equipment in the business platform is damaged or does not have this monitoring function, the specific data cannot be returned and the data of the preset standard attribute needs to be left blank.

[0053] Optionally, if the currently received data type includes text, obtaining data corresponding to a preset standard attribute in the text and determining the target data further includes:

[0054] If the currently received data type includes a picture stream, the picture stream is converted into binary data;

[0055] Store the currently received binary data into the database.

[0056] In this embodiment, converting the image stream into binary data is a prior art, so it will not be described in detail.

[0057] Optionally, if the currently received data type includes text, obtaining data corresponding to a preset standard attribute in the text and determining the target data further includes:

[0058] If the currently received data type includes a video stream, the video is pulled from the website corresponding to the video stream;

[0059] Store the currently received video in the database.

[0060] In this embodiment, a crawler script may be used to pull videos from a URL according to a video stream. The crawler script is a prior art and will not be described in detail.

[0061] Optionally, checking the data type of the currently received data further includes:

[0062] Encrypt and verify the currently received data;

[0063] If the currently received data is encrypted data, the encrypted data is decrypted.

[0064] In this embodiment, it is necessary to unify the encryption method with the manufacturer and use the unified encryption method to encrypt the data. The unified encryption method is conducive to encryption management of data and improves data security.

[0065] Optionally, the encryption method includes key encryption or code encryption.

[0066] In this embodiment, data is uniformly encrypted through key encryption or code encryption, which is beneficial to encryption management of data and improves data security.

[0067] Optionally, after the fused data is obtained, it can also be used to monitor water conservancy projects. For example, in the water conservancy industry, it is necessary to monitor the rainfall at water and rainfall stations, and add weather forecasts to predict the flood season.

[0068] Based on the above, we first determine the selection of flood forecasting model, and combine the characteristics of supervised pre-training ability for system input data in deep learning model theory, and use RBF neural network algorithm to repeatedly train the water level and flow data simulation forecasting behavior required for flood forecasting, so as to improve the forecasting accuracy of the system calculation model. Among them, the RBF neural network algorithm is not only applied to a single module, but also to various demand business modules in the entire engineering standardized operation system, including the highly correlated behavioral operations generated by system users when applying the system, which will be predicted by the system through the algorithm to the user's preferences and business scenario selection.

[0069] After training the RBF neural network with fusion data, the corresponding flood prediction model can be obtained. The flood prediction model can calculate the impending, short-term, medium-term and long-term reservoir inflows to facilitate the prediction of the flood period.

[0070] like Figure 2 As shown, the present invention provides a multi-source heterogeneous data fusion system, comprising:

[0071] A monitoring module is used to monitor a preset port in real time; wherein the preset port is used to receive data sent by each monitoring device;

[0072] A verification module is used to verify the data type of the currently received data;

[0073] A target data determination module is used to obtain data corresponding to a preset standard attribute in the text to determine the target data if the currently received data type includes text;

[0074] The multi-source heterogeneous data fusion module is used to perform multi-source heterogeneous data fusion on each target data currently received, determine the fused data, and store the fused data in the database.

[0075] Optionally, the target data determination module is specifically used to:

[0076] Get the data corresponding to each preset field in the text;

[0077] The acquired data corresponding to each preset field is used as the target data.

[0078] Optionally, the target data determination module is further used to:

[0079] If the data corresponding to any preset field in this article is missing, the corresponding missing data is requested from the business platform corresponding to the text data.

[0080] Optionally, the system further includes an image processing module, specifically configured to:

[0081] If the currently received data type includes a picture stream, the picture stream is converted into binary data;

[0082] Store the currently received binary data into the database.

[0083] Optionally, the system further includes a video processing module, specifically configured to:

[0084] If the currently received data type includes a video stream, the video is pulled from the website corresponding to the video stream;

[0085] Store the currently received video in the database.

[0086] Optionally, the system further includes an encryption module, specifically configured to:

[0087] Encrypt and verify the currently received data;

[0088] If the currently received data is encrypted data, the encrypted data is decrypted.

[0089] An embodiment of the present invention also provides a computing device, including a memory, a manager, and a program stored in the memory and running on the manager. When the manager executes the program, it implements some or all steps of the above-mentioned multi-source heterogeneous data fusion method.

[0090] Among them, the computing device can be a computer, and correspondingly, its program is computer software. The above parameters and steps in the computing device of the present invention can refer to the parameters and steps in the embodiment of a multi-source heterogeneous data fusion method mentioned above, and will not be repeated here.

[0091] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.

[0092] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0093] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-source heterogeneous data fusion method, It is characterized in that include: Monitor the preset port in real time; wherein the preset port is used to receive data sent by each monitoring device; Verify the data type of the currently received data; If the currently received data type includes text, then obtaining data corresponding to a preset standard attribute in the text to determine the target data; The currently received target data are subjected to multi-source heterogeneous data fusion, fused data is determined, and the fused data is stored in a database.

2. The method according to claim 1, It is characterized in that Acquiring data corresponding to a preset standard attribute in the text and determining target data includes: Obtain data corresponding to each preset field in the text; The acquired data corresponding to each of the preset fields is used as the target data.

3. The method according to claim 2, It is characterized in that Acquiring data corresponding to each preset field in the text also includes: If the data corresponding to any of the preset fields in the text is missing, the corresponding missing data is requested from the business platform that sent the text.

4. According to the method of claim 1, It is characterized in that The method further includes: If the currently received data type includes a picture stream, converting the picture stream into binary data; Store the currently received binary data into the database.

5. According to the method of claim 1, It is characterized in that The method further includes: If the currently received data type includes a video stream, then pull the video from the website corresponding to the video stream; Store the currently received video in the database.

6. The method according to any one of claims 1 to 5, It is characterized in that The method further includes: Encrypt and verify the currently received data; If the currently received data is encrypted data, the encrypted data is decrypted.

7. According to the method of claim 6, It is characterized in that The encryption method includes key encryption or code encryption.

8. A multi-source heterogeneous data fusion system, It is characterized in that include: A monitoring module, used for monitoring a preset port in real time; wherein the preset port is used for receiving data sent by each monitoring device; A verification module is used to verify the data type of the currently received data; A target data determination module, configured to obtain data corresponding to a preset standard attribute in the text to determine the target data if the currently received data type includes text; The multi-source heterogeneous data fusion module is used to perform multi-source heterogeneous data fusion on each target data currently received, determine fused data, and store the fused data in a database.

9. A computing device comprising a memory, a processor, and a program stored in the memory and running on the processor, It is characterized in that When the processor executes the program, the steps of a multi-source heterogeneous data fusion method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the terminal device, the terminal device executes the steps of a multi-source heterogeneous data fusion method as described in any one of claims 1 to 7.