Multi-source heterogeneous data fusion method and system

Abstract

The invention relates to the technical field of information fusion, in particular to a multi-source heterogeneous data fusion method and system. The multi-source heterogeneous data fusion system comprises an intelligent contract automation platform, an ultimate data fusion module, a data packet data acquisition module, a data packet data processing module and a file format conversion node expansion module. According to the method, the uploaded data packet is received in real time through the intelligent contract automation platform, the source of the data packet is verified, the intelligent contract condition met by the data packet is obtained according to the source of the data packet, and the corresponding data processing logic and protocol under the intelligent contract condition are triggered to pre-process the data packet; according to the data fusion method and system, the preprocessed data are transmitted to the ultimate data fusion end, and the final data fusion result is output, so that the validity and accuracy of the data are well ensured, the pressure brought to data integration, management and analysis due to the heterogeneity of the data is relieved, and the probability of occurrence of information islands and data isolation phenomena is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of information fusion, and more specifically, it relates to a method and system for multi-source heterogeneous data fusion. Background Art

[0002] Since multi-source heterogeneous data comes from different data sources and has different formats and attributes, it is very likely that there are situations of incomplete data, inaccurate data, and inconsistent data, making it difficult to directly integrate them into effective and accurate data. In addition, since these data are usually represented in different structures and semantics, there is heterogeneity between them. This data heterogeneity poses great challenges to data integration, management, and analysis, and may lead to the emergence of information islands and data isolation phenomena. Summary of the Invention

[0003] In view of the above problems, the present application provides a method and system for multi-source heterogeneous data fusion, which perform data fusion on data packets from different sources to obtain a data fusion result, well ensuring the effectiveness and accuracy of the data and alleviating the pressure brought by data heterogeneity to data integration, management, and analysis.

[0004] A method for multi-source heterogeneous data fusion includes: S1: Based on blockchain technology, establish an intelligent contract automation platform, which includes a transmission node, an unpacking node, and a file format conversion node. The number of types of data packets is the same as the initial number of unpacking nodes and file format conversion nodes; S2: The transmission node receives the uploaded data packet α in real time, verifies the source of the data packet α, obtains the intelligent contract condition satisfied by the data packet α according to the source of the data packet α, triggers the unpacking logic under this intelligent contract condition, executes the unpacking protocol, obtains the file format type of the data packet α, and obtains the intelligent contract condition satisfied by the data packet α according to this file format type, triggers the file format conversion logic under this intelligent contract condition, executes the file format conversion protocol, obtains a primary data packet, and transmits the primary data packet to the ultimate data fusion end; S3: Determine whether the number of primary data packets reaches a preset number of data packets. If the number of primary data packets reaches the preset number of data packets, then fuse these data packets together through the ultimate data fusion end and output the final data fusion result; otherwise, continue to determine whether the number of primary data packets reaches the preset number of data packets.

[0005] As a preferred aspect of the present application, for the unpacking logic and file format conversion logic in step S2, it specifically includes: Unpacking logic: Transmit the data packet α to the corresponding unpacking node, and the unpacking node decompresses the transmitted data packet and identifies the file format of the data packet after decompression; File format conversion logic: Transmit the data packets after the unpacking protocol is executed to the corresponding file format conversion nodes. The file format conversion nodes perform file format conversion on the transmitted data packets to obtain primary data packets, and then transmit the primary data packets to the ultimate data fusion end.

[0006] As a preferred aspect of this application, it further includes: further optimization of the unpacking logic in step S2, specifically as follows: S2.1: Denote the number of current unpacking nodes as to obtain an unpacking node set where represents any unpacking node in the unpacking node set. The unpacking node unpacks the respective transmitted data packets to obtain an unpacked data packet set where represents any data packet in the data packet set, and the corresponding unpacking node of the data packet is ; S2.2: Calculate the number of bytes occupied by the data packet and convert the result to a preset unit to obtain a data packet size set where represents any data packet size in the data packet size set, is the corresponding size of the data packet , take , is the smallest data packet size in the data packet size set . Denote the data packet with the data packet size of as the data packet ; S2.3: Put the for which " " holds into a data packet size temporary storage set , is the preset data size, where represents any data packet size in the data packet size temporary storage set; S2.4: Based on the original unpacking logic and the data packet size temporary storage set, obtain its corresponding file format conversion node set and a data packet temporary storage set , where the corresponding file format conversion node of the data packet is , is the corresponding size of the data packet ; S2.5: Let " ", is the size of the data packet ​ With respect to the data packet size to obtain a multiple set , let , where denotes taking the ceiling of the real number , is an integer obtained by taking the ceiling of the real number to obtain an integer set ; S2.6: Expand the file format conversion node into identical functional file format conversion nodes, and use the multi - volume compression technology for the data packet to separately compress the files inside the data packet into pieces, label each separately compressed package as a sub - volume, and the expanded file format conversion node set form and the compressed package set form after multi - volume compression are as follows: ; For the set , where the expanded file format conversion node set , denotes any expanded file format conversion node set in the expanded file format conversion node set , is denoted as any expanded file format conversion node in the expanded file format conversion node set ; For the set K, where the multi - volume compressed package set , denotes any multi - volume compressed package set in the multi - volume compressed package set , is denoted as any compressed package in the multi - volume compressed package set ; S2.7: Transmit the compressed packages inside the multi - volume compressed package set to the corresponding file format conversion node inside the expanded file format conversion node set .

[0007] As a preferred aspect of this application, the format types of the files inside the data packet can be TIFF, ASM, XLSX, etc.

[0008] As a preferred aspect of this application, the data packet is uploaded to the transmission node through a wireless transmission method.

[0009] As a preferred aspect of the present application, the compressed data packet is generally decompressed by one of the following decompression tools , , .

[0010] Provided is a multi-source heterogeneous data fusion system, including: An intelligent contract automation platform, which is used to preprocess the data packet; An ultimate data fusion module, which is used to perform ultimate data fusion on the data packet.

[0011] As a preferred aspect of the present application, it further includes: A data packet data acquisition module, which is used to acquire data packet data; A data packet data processing module, which is used to process data packet data; A file format conversion node expansion module, which is used to expand file format conversion nodes.

[0012] The present invention has the following advantages: 1. The present invention receives the uploaded data packet in real time through the intelligent contract automation platform, verifies the source of the data packet, obtains the intelligent contract conditions satisfied by the data packet according to the source of the data packet, triggers the corresponding data processing logic and protocol under the intelligent contract conditions to preprocess the data packet, transmits the preprocessed data to the ultimate data fusion end, and outputs the final data fusion result, which well guarantees the validity and accuracy of the data, alleviates the pressure brought to data integration, management and analysis due to the heterogeneity of the data, and reduces the probability of the appearance of information islands and data isolation phenomena.

[0013] 2. The present invention acquires the data of the data packet through the data packet data acquisition module, processes the data packet data through the data packet data processing module, and expands the file format conversion node according to the processed data packet data by the file format conversion node expansion module, effectively solving the problem of data packet blockage that may occur during the process of receiving and processing data packets, shortening the time for each data packet to convert the file format, and improving the data fusion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the structures shown in these drawings without creative efforts.

[0015] Figure 1 This is a schematic flowchart of a multi-source heterogeneous data fusion method in Embodiment 1 of the present invention.

[0016] Figure 2 This is a schematic structural diagram of a multi-source heterogeneous data fusion system in Embodiment 2 of the present invention. Specific embodiments

[0017] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further details some embodiments of the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. However, those of ordinary skill in the art can understand that in various embodiments of the present application, many technical details are proposed to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0018] Embodiment 1: Refer to Figure 1 , Embodiment 1 of the present invention provides a multi-source heterogeneous data fusion method, including: S1: Based on blockchain technology, establish an intelligent contract automation platform, which includes a transmission node, an unpacking node, and a file format conversion node, and the number of types of data packets is the same as the initial number of unpacking nodes and file format conversion nodes; For step S1, it should be further explained that blockchain technology is a decentralized and disintermediated technology, which has characteristics such as information immutability, openness and transparency, and collective maintenance, and is a technology for a reliable database; Among them, an intelligent contract can automatically execute the content of the contract. When the preset conditions are met, corresponding operations can be automatically executed. By identifying the intelligent contracts corresponding to a large number of data packets, the corresponding data processing logics and protocols under the conditions of the intelligent contracts corresponding to each data packet can be automatically triggered to preprocess the data packet.

[0019] Based on blockchain technology, establishing an intelligent contract automation platform can well ensure the validity and accuracy of data, relieve the pressure on data integration, management and analysis caused by the heterogeneity of data, and reduce the probability of information islands and data isolation phenomena.

[0020] S2: The transmission node receives the uploaded data packet α in real time, verifies the source of the data packet α, obtains the smart contract conditions satisfied by the data packet α according to the source of the data packet α, triggers the unpacking logic under the smart contract conditions, executes the unpacking protocol, obtains the file format type of the data packet α, and obtains the smart contract conditions satisfied by the data packet α according to the file format type, triggers the file format conversion logic under the smart contract conditions, executes the file format conversion protocol, obtains the primary data packet, and transmits the primary data packet to the ultimate data fusion end; For step S2, it should be further noted that the data packet is uploaded to the transmission node through a wireless transmission method; the format types of the internal files of the data packet can be TIFF, ASM, XLSX, etc.; the transmission protocols of any form of data packets with the same source are the same.

[0021] S3: Determine whether the number of primary data packets reaches the preset number of data packets. If the number of primary data packets reaches the preset number of data packets, then fuse these data packets together through the ultimate data fusion end and output the final data fusion result; otherwise, continue to determine whether the number of primary data packets reaches the preset number of data packets.

[0022] For the unpacking logic and file format conversion logic in step S2, specifically include: Unpacking logic: Transmit the data packet α to the corresponding unpacking node, and the unpacking node decompresses the transmitted data packet and identifies the file format of the data packet after decompression; File format conversion logic: Transmit the data packet after the unpacking protocol is executed to the corresponding file format conversion node, and the file format conversion node performs file format conversion on the transmitted data packet to obtain the primary data packet, and transmits the primary data packet to the ultimate data fusion end.

[0023] For the decompression process of the data packet, it should be further noted that the compressed data packet is generally decompressed by one of the decompression tools in 、 、 .

[0024] It also includes: Further optimization of the unpacking logic in step S2 is as follows: S2.1: Record the number of current unpacking nodes as , obtain the unpacking node set , where represents any unpacking node in the unpacking node set, and the unpacking node decompresses the respective transmitted data packets to obtain the decompressed data packet set , where represents any data packet in the data packet set, and the data packet The corresponding unpacking node is ; S2.2: Calculate the number of bytes occupied by the data packet and convert the result to a preset unit to obtain a data packet size set , where represents any data packet size within the data packet size set, is the corresponding size of the data packet , take , is the smallest data packet size in the data packet size set , and mark the data packet with a data packet size of as data packet ; For step S2.2, it should be further noted that the preset unit is set manually according to the actual situation of the data packet, and the preset unit generally includes KB, MB, GB, etc.

[0025] S2.3: Put the " holds into the data packet size temporary storage set , is the preset data size, where represents any data packet size within the data packet size temporary storage set; For step S2.3, it should be further noted that the preset data size is set manually according to the actual situation of the data packet; Among them, the " is satisfied is put into the data packet size temporary storage set, and the file format conversion node corresponding to its data packet is expanded to prevent data packet congestion when converting the file format of the data packet, so that the internal file of the obtained data packet is inaccurate, effectively solving the possible data packet congestion problem during the process of receiving and processing data packets, shortening the time for converting the file format of each data packet, and improving the data fusion efficiency.

[0026] S2.4: Based on the original unpacking logic and the data packet size temporary storage set, obtain the corresponding file format conversion node set and the data packet temporary storage set , where the file format conversion node corresponding to the data packet is , is the corresponding size of the data packet ; S2.5: Let " ", is the data packet size With respect to the data packet size a multiple of, to obtain a set of multiples , let , where denotes taking the ceiling of the real number , is the real number the integer obtained by taking the ceiling, to obtain a set of integers ; For step S2.5, it should be further noted that by calculating the size of the data packet size with respect to the size of the data packet size a multiple of, taking the ceiling of the multiple to obtain the number of data packets to be split and compressed, accurately obtaining the split and compression information of each data packet, facilitating the precise splitting of the data packet and realizing the file format conversion process for the split and compressed packets simultaneously, effectively improving the efficiency of the file format conversion within the data packet .

[0027] S2.6: Expand the file format conversion node into identical functional file format conversion nodes, and use the split and compression technology for the data packet to separately compress the files inside the data packet into pieces, label each separately compressed packet as a split volume, and the expanded set of file format conversion nodes form and the set of compressed packets after split and compression form are: ; For the set , where the expanded set of file format conversion nodes , denotes an arbitrarily expanded set of file format conversion nodes in the expanded set of file format conversion nodes , denotes an arbitrarily expanded file format conversion node in the expanded set of file format conversion nodes ; For the set K, where the set of split and compressed packets , denotes an arbitrarily selected set of split and compressed packets in the set of split and compressed packets , denotes an arbitrarily selected compressed packet in the set of split and compressed packets ; For step S2, it should be further noted that the file format conversion node , it is necessary to endow the originally representative single file format conversion node with a new meaning and change it to an extended set of file format conversion nodes; among which the data packet , it is necessary to endow the originally representative single data packet with a new meaning and change it to the set of split volume compressed packages corresponding to the data packet.

[0028] S2.7: Transfer the compressed packages inside the set of split volume compressed packages to the extended set of file format conversion nodes corresponding to it inside the file format conversion nodes .

[0029] The present invention receives the uploaded data packet in real time through the intelligent contract automation platform, verifies the source of the data packet, obtains the intelligent contract conditions satisfied by the data packet according to the source of the data packet, triggers the corresponding data processing logic and protocol under the intelligent contract conditions to preprocess the data packet, transmits the preprocessed data to the ultimate data fusion end, and outputs the final data fusion result, which well guarantees the validity and accuracy of the data, alleviates the pressure brought to data integration, management and analysis due to the heterogeneity of the data, and reduces the probability of the appearance of information islands and data isolation phenomena.

[0030] The present invention obtains the data of the data packet through the data packet data acquisition module, processes the data of the data packet through the data packet data processing module, and expands the file format conversion node according to the processed data of the data packet by the file format conversion node expansion module, effectively solving the problem of data packet blockage that may occur during the process of receiving and processing the data packet, shortening the time for each data packet to convert the file format, and improving the data fusion efficiency.

[0031] Embodiment 2: Refer to Figure 2 , Embodiment 2 of the present invention provides a multi-source heterogeneous data fusion system, including: An intelligent contract automation platform, which is used to preprocess the data packet; An ultimate data fusion module, which is used to perform ultimate data fusion on the data packet.

[0032] Refer to Figure 2 , Embodiment 2 of the present invention provides a multi-source heterogeneous data fusion system, further including: A data packet data acquisition module, which is used to acquire data packet data; A data packet data processing module, which is used to process data packet data; File format conversion node expansion module, which is used to expand file format conversion nodes.

[0033] It should be understood that those of ordinary skill in the art can make improvements or transformations according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention. The parts not described in detail in this specification belong to the prior art well known to those of ordinary skill in the art.

Claims

1. A multi-source heterogeneous data fusion method, characterized in that: include: S1: Based on blockchain technology, a smart contract automation platform is established, which includes transmission nodes, unpacking nodes and file format conversion nodes. The number of data packet types is consistent with the initial number of unpacking nodes and file format conversion nodes; S2: The transmission node receives the uploaded data packet α in real time, verifies the source of the data packet α, obtains the smart contract conditions satisfied by the data packet α according to the source of the data packet α, triggers the unpacking logic under the smart contract conditions, executes the unpacking protocol, obtains the file format type of the data packet α, and obtains the smart contract conditions satisfied by the data packet α according to the file format type, triggers the file format conversion logic under the smart contract conditions, executes the file format conversion protocol, obtains the primary data packet, and transmits the primary data packet to the ultimate data fusion end; S3: Determine whether the number of primary data packets reaches the preset number of data packets. If so, merge these data packets together through the ultimate data fusion terminal and output the final data fusion result; otherwise, continue to determine whether the number of primary data packets reaches the preset number of data packets.

2. A multi-source heterogeneous data fusion method as claimed in claim 1, characterized in that: The unpacking logic and file format conversion logic in step S2 specifically include: Unpacking logic: The data packet α is transmitted to the corresponding unpacking node. The unpacking node decompresses the transmitted data packet and identifies the file format of the data packet after decompression. File format conversion logic: The data packet after the unpacking protocol is executed is transmitted to the corresponding file format conversion node. The file format conversion node converts the file format of the transmitted data packet to obtain the primary data packet, and transmits the primary data packet to the ultimate data fusion end.

3. A multi-source heterogeneous data fusion method as claimed in claim 2, characterized in that: Also includes: The further optimization of the unpacking logic in step S2 is as follows: S2.1: Record the number of currently unpacked nodes as , get the unpacked node set ,in Indicates any unpacked node in the unpacked node set, unpacked node Decompress each transmitted data packet to obtain a set of decompressed data packets ,in Indicates any data packet in the data packet set. The corresponding unpacking node is ; S2.2: Calculation of data packets The number of bytes occupied by the packet is converted into the preset unit to obtain the packet size set. ,in Indicates any packet size in the packet size set, For data packets Corresponding size, take , Packet size set The packet size with the smallest value is The data packets are recorded as data packets ; S2.3: " "Established Put the data packet size into temporary storage collection , is the preset data size, where Indicates the size of any data packet in the temporary storage set of data packets; S2.4: Based on the original unpacking logic and the temporary storage set of data packet sizes, the corresponding file format conversion node set is obtained And the data packet temporary storage collection , where data packets The corresponding file format conversion node is , For data packets Corresponding size; S2.5: Order ”, For data packets size Relative to the data packet size Multiples of , get multiples set ,make ,here Represents real numbers Take the integer to the largest value, is a real number Get the integer after taking the integer to the largest, and get the integer set ; S2.6: Convert file format to node Expand to The same file format conversion node converts the data packet Using volume compression technology, the files in the data package are compressed separately into Each individual compressed package is marked as a sub-volume, and the expanded file format conversion node set The format and the compressed package collection after volume compression The form is: ； For collections , where the file format conversion node expands the collection , Represents the expanded set of file format conversion nodes The extended set of nodes for arbitrary file format conversion in Represents an expanded set of file format conversion nodes Any expanded file format conversion node; For set K, the volume compression package set , Represents a collection of sub-volume compressed packages Any collection of sub-volume compressed packages, Represented as a collection of sub-volume compressed packages Any compressed package in it; S2.7: Assemble the sub-volume compression packages Internal compression package Transfer to its corresponding file format conversion node expansion set Internal file format conversion node .

4. A multi-source heterogeneous data fusion method as claimed in claim 3, characterized in that: include: The format type of the files inside the data package can be TIFF, ASM, XLSX, etc.

5. A multi-source heterogeneous data fusion method as claimed in claim 4, characterized in that: include: The data packet is uploaded to the transmission node via wireless transmission.

6. A multi-source heterogeneous data fusion method as claimed in claim 5, characterized in that: include: Compressed data packets are usually , , Use one of the decompression tools in to decompress the file.

7. A multi-source heterogeneous data fusion system, characterized in that: include: Smart contract automation platform, which is used to pre-process data packets; The ultimate data fusion module is used to perform ultimate data fusion on data packets.

8. A multi-source heterogeneous data fusion system as claimed in claim 7, characterized in that: Also includes: A data packet data acquisition module, the data packet data acquisition module is used to acquire data packet data; A data packet data processing module, the data packet data processing module is used to process data packet data; A file format conversion node expansion module is used to expand the file format conversion node.

Images