Multi-source heterogeneous data distributed fusion method

Through logarithmic transformation, range-based sharding, Edge for Cloud model and parallel processing, data preprocessing complexity and consistency problems in multi-source heterogeneous data fusion are solved, and efficient and real-time data fusion is achieved. Data sources adapted to dynamic changes.

CN118916411AActive Publication Date: 2024-11-08SHANGHAI JINRON DIGITS TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411035308.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

When facing large-scale, multi-source and heterogeneous data, the data preprocessing is complex and time-consuming, and it is difficult to ensure the consistency and integrity of data in a distributed environment. Traditional consistency algorithms Inefficient, it cannot meet the real-time processing needs, and the heterogeneous data format and semantics are large, the format conversion efficiency is low, making it difficult to adapt to dynamically changing data sources.

Method used

Multi-source heterogeneous data distributed fusion method is adopted, including data preprocessing through logarithmic transformation, data sharding uses a range-based sharding method, multi-level caching adopts the Edge for Cloud model, caches tags through distributed edge nodes, and parallelizes Processing and consistency verification, dynamically generate heterogeneous data mapping rules, and optimize mapping rules using machine learning.

Benefits of technology

It improves the efficiency and consistency of data processing, keeps the dimension of the data unchanged, can quickly retrieve and extract required data information, adapt to dynamically changing data sources, controls the overhead of NP-hard problems, and meets the real-time processing needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916411A_ABST
    Figure CN118916411A_ABST
Patent Text Reader

Abstract

The invention provides a distributed fusion method for multi-source heterogeneous data, and aims to improve the data processing speed, the data consistency and reliability and the heterogeneous data mapping and conversion efficiency, the method comprises four steps of data preprocessing, data fragmentation, multi-level caching and distributed fusion, in the data preprocessing stage, the multi-level caching and the distributed fusion are carried out; the data format is unified through logarithm ratio transformation; in the data fragmentation stage, the data are refined in a range-based fragmentation mode; in the multi-level cache stage, an Edge for Cloud model is applied to realize cooperative work of edge nodes and cloud computing power; in the data fusion stage, a unified data view is generated through data extraction, data matching and data merging. The method has the beneficial effects that the method can be widely applied to a large-scale data processing scene, and an efficient and reliable data fusion scheme is provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data and edge computing, and in particular to a distributed fusion method for multi-source heterogeneous data. Background Art

[0002] In the era of big data, the multi-source and heterogeneous nature of data has become a major challenge in data processing and analysis. Data comes from a variety of sources, including structured data, semi-structured data, and unstructured data, which are usually distributed on different systems and platforms and have different formats and semantics. The existence of such multi-source heterogeneous data makes it difficult for traditional data processing methods to cope with it, especially in application scenarios that require real-time processing and fusion of such data.

[0003] Structured data is usually stored in a relational database, which is characterized by a strict schema, fixed data types and clear relationships. Common sources of structured data include business systems of enterprises, financial transaction records, etc. Semi-structured data contains certain structural information, but it is not as strict as structured data, such as data in XML and JSON formats. They are widely present in scenarios such as Web data and data returned by APIs. Unstructured data has no fixed structure at all, and is common in data types such as text, images, and videos. These data come from a wide range of sources, such as social media, sensor data, log files, etc.

[0004] In order to effectively utilize these multi-source heterogeneous data, data fusion technology has emerged. Data fusion aims to integrate data from different sources and formats to form a unified view for subsequent analysis and mining. However, existing data fusion technology still faces many challenges when facing large-scale, multi-source, and heterogeneous data. First, data preprocessing is a complex and time-consuming process, including data cleaning, format conversion, and outlier processing. Secondly, in a distributed environment, how to ensure data consistency and integrity is a key issue in data fusion. Although traditional distributed consistency algorithms such as Paxos and Raft can solve the consistency problem to a certain extent, they are inefficient when processing large-scale data and cannot meet the needs of real-time processing. Thirdly, the formats and semantics of heterogeneous data are very different. How to effectively map and convert is another major difficulty in realizing data fusion. Existing format conversion methods are usually inefficient and difficult to adapt to dynamically changing data sources. Summary of the invention

[0005] In view of the above problems, the present invention aims to provide a distributed fusion method for multi-source heterogeneous data to solve the problems raised in the above background technology.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] 1. Distributed fusion method of multi-source heterogeneous data, including: data preprocessing, data sharding, multi-level caching and distributed fusion.

[0008] Furthermore, the data preprocessing defines that the data space contains m groups of data elements, and the m groups of data elements are logarithmically transformed on all components simultaneously, but the benchmark is not a single component, but the geometric mean of all components, which is:

[0009]

[0010] Among them, x i is the i-th initial data element, z i is the i-th data element.

[0011] Furthermore, the data sharding step is specifically as follows: define the data space as Z, the data space includes m data, each data includes n sets of labels, if the label does not contain any elements, the data space is not default, recorded as null, then there is

[0012]

[0013] , where z1 is the first group of data elements in data space Z, z2 is the second group of data elements in data space Z, and z m is the mth group of data elements in the data space Z, is the first set of labels for z1, is the second set of labels for z1, is the nth group label of z1, is the first set of labels for z2, is the second set of labels for z2, is the nth group label of z2, For z m The first set of labels, For z m The second set of labels, For z m The nth group of labels, a range-based sharding method is proposed to shard the data elements with labels in the data space, and there are;

[0014]

[0015] Among them, low_1 is the lower limit of the first group range, up_1 is the upper limit of the first group range, low_2 is the lower limit of the second group range, up_2 is the upper limit of the second group range, low_q is the lower limit of the qth group range, up_q is the upper limit of the qth group range, For data elements under different scope constraints The labels belong to the range [low_1,up_1], [low_2,up_2], [low_q,up_q] respectively, and have:

[0016]

[0017] Among them, Θ is an empty set, and there exists This is because the data elements have multiple sets of labels. In the range-based sharding method, the selection of data elements is blurred, and the attribute value of the label is more important. The mapping relationship between data elements and labels is one-to-many, and there are:

[0018]

[0019] Furthermore, the multi-level caching step is specifically as follows: multi-level cache is set at each node to accelerate data reading and writing operations, and it is proposed to use the Edge for Cloud model to cache data. On the basis of sharding the data elements with labels in the data space in a range-based sharding manner, q groups of subspaces are formed, each group of spaces contains independent labels, and the labels are cached through distributed edge nodes. Each label is regarded as a node, and the data space contains q groups of nodes for caching labels within the range, and the data space contains 1 cloud computing power, whose main function is to provide computing power and storage support for q edge nodes, which specifically includes the following:

[0020] In the framework of the q group subspace, there are labels: Note (1) ,note (2) ,…,note (q) are the labels in the first classification group, the labels in the second classification group, and the labels in the qth classification group respectively. The computing power set of the edge node is defined as computingpower=

[0021] {computingpower1,computingpower2,…,computingpower q}, among which computingpower1, computingpower2,…,computingpower q They are the computing power of the first edge node, the computing power of the second edge node, and the computing power of the qth edge node. The computing power of the cloud computing power is defined as COMPUTINGPOWER CLOUD ,have

[0022] Furthermore, in order to enhance the competitiveness of the data space, an overhead control mechanism is proposed to quantitatively control the cache of the data space. The overhead control mechanism includes:

[0023] (1) Edge node overhead and cloud overhead

[0024] The data attributes of edge nodes are defined as:

[0025] {Edge:computingpower,storagespace,offloading,cost edge ,cost cloud}

[0026] Among them, Edge is the edge node, computingpower is the computing power, storagespace is the storage space, offloading is the computing offloading, and the label will be offloaded to the specified edge node for caching. edge is the edge node overhead, cost cloud It is the cloud computing power overhead. The node overhead includes two situations: 1. When the node computing power or storage space supports the cache volume of the current computing offload, the node overhead belongs to the current edge node; 2. When the node computing power or storage space cannot meet the cache volume of the current computing offload, it is necessary to rent an edge node with idle node computing power or storage space in the data space for caching;

[0027] (2) Service selection for edge nodes

[0028] There are three ways to select services for edge nodes. If the computing power of the current edge node is sufficient but the storage space cannot provide services for the tag, the tag will wait online to complete the service; if the computing power of the current edge node is insufficient but the storage space can provide services for the tag, it can cooperate with the cloud computing power to participate in the cache service; if the computing power of the current edge node is insufficient and the storage space cannot provide services for the tag, it can call other edge nodes with computing power and storage space to collaborate on the cache task;

[0029] For label online waiting, there are:

[0030]

[0031] C total ={1:Load1,2:Load2,…|EMPTY_ROOM}

[0032] Among them, ρ is the service intensity, μ is the service rate, λ is the unloading arrival rate, For the tag note * Unloading, computing power * is for the tag note* The edge node of the service, Bandwidyh is the network communication bandwidth, T cache is the cache time, is the total execution time when the tag is waiting online, C total is the total storage space of the edge node, Load1 is the first unloaded amount, Load2 is the second unloaded amount, and EMPTY_ROOM is the remaining storage space. but Enter the waiting service sequence of the edge node and complete the cache service in order;

[0033] For collaborative cloud computing power to participate in cache services, there are:

[0034]

[0035]

[0036] COST (2) =cost edge +cost cloud

[0037] in, is the total execution time of the collaborative cloud computing power participating in the cache service, The storage space for cloud computing power, COST (2) The total cost of coordinating cloud computing power to participate in caching services;

[0038] For other edge nodes with computing power and storage space, we coordinate caching services:

[0039]

[0040] COST (3) =cost edge +cost edge+

[0041] Among them, edge+ is the edge node that participates in the coordination of cache services, computingpower + For edge+ computing power, EMPTYROOM + The remaining space is edge+, The total execution time when coordinating cache services for other edge nodes with computing power and storage space, cost edge+ Cost of leasing edge+, COST (3) The total cost of coordinating caching services for other edge nodes with computing power and storage space;

[0042] (3) Parallel processing uses parallel processing technology to allow each node to independently process the sharded data in parallel, using multi-threading or multi-process technology to accelerate processing, using message queues to achieve communication and data synchronization between nodes, and processing different data fragments at the same time to shorten processing time;

[0043] (4) Consistency Check After data processing is completed, the data of each node is checked for consistency. The Raft protocol is used to perform consistency checks between nodes to ensure data consistency.

[0044] Furthermore, the heterogeneous data mapping mechanism dynamically generates heterogeneous data mapping rules according to the labels of data elements, including:

[0045] W=(zb)note T (note note T ) -1

[0046]

[0047] Where W is the weight matrix, z is the data element matrix, b is the bias vector, note is the label vector, and z i is the i-th group of data elements, note j For z j Tags, b j For z j bias, use machine learning technology to learn from historical data and continuously optimize the mapping rules.

[0048] Furthermore, the data fusion includes: data extraction, data matching and data merging. Data extraction is to extract the data to be fused from various data sources; data matching is to match the extracted data to identify the same or similar data items; data merging is to merge the matched data items to generate a unified data view.

[0049] Beneficial effects of the present invention:

[0050] 1. The present invention adopts a logarithmic ratio transformation to pre-process the initial data elements. The traditional common logarithmic ratio transformation method is usually to compare the elements in a component vector with another selected reference element, and then take the natural logarithm of the ratio. This transformation will cause the data to lose its original proportional properties and can handle common scale problems and proportional dependence in proportional data. The logarithmic ratio transformation method provided by the present invention is to perform logarithmic ratio transformation on all components at the same time, but the reference is not a single component, but the geometric mean of all components. The advantage is that it keeps the dimension of the data unchanged and does not need to select a specific reference component. Therefore, it is more convenient and consistent when processing multivariate component data.

[0051] 2. The present invention proposes range-based data slicing, in which the data is characterized by a one-to-many mapping relationship. This means that in data slicing, the selection of data elements is non-independent, while the selection of labels is unique. In the range-based slicing method, the selection of data elements is blurred, and more emphasis is placed on the attribute value of the label, thereby enhancing the effect of data classification, being able to quickly retrieve the required data information within a specified range, and being able to extract data elements.

[0052] 3. The present invention proposes to cache data using the Edge for Cloud model. Based on the range-based sharding method for sharding the data elements with labels in the data space, q groups of subspaces are formed. Each group of spaces contains independent labels. The labels are cached through distributed edge nodes. An overhead control mechanism is proposed to quantitatively control the cache of the data space. In the service selection link of the edge node, the distributed data is processed. The situations of data queuing, insufficient computing power, and insufficient storage space are taken into consideration. A model is provided in which the labels are waiting online, the cloud computing power is coordinated to participate in the caching service, and other edge nodes with computing power and storage space coordinate the caching service. The overhead is used as a constraint to control the NP-hard problem within the range of feasible solution space. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The invention is further illustrated by the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0054] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 The present invention will be further described in conjunction with the following examples.

[0057] See also Figure 1 The present invention aims to provide a distributed fusion method for multi-source heterogeneous data, which is characterized by comprising the following steps: data preprocessing, data sharding, multi-level caching and distributed fusion.

[0058] Specifically, the data preprocessing defines that the data space contains m groups of data elements, and performs logarithmic ratio transformation on all components of the m groups of data elements simultaneously, but the benchmark is not a single component, but the geometric mean of all components, which is:

[0059]

[0060] Among them, x i is the i-th initial data element, z i is the i-th data element.

[0061] Specifically, the data sharding step is as follows: define the data space as Z, the data space includes m data, each data includes n sets of labels, if the label does not contain any elements, the data space is not default, recorded as null, then there is

[0062]

[0063] Where z1 is the first group of data elements in data space Z, z2 is the second group of data elements in data space Z, and z m is the mth group of data elements in the data space Z, is the first set of labels for z1, is the second set of labels for z1, is the nth group label of z1, is the first set of labels for z2, is the second set of labels for z2, is the nth group label of z2, For z m The first set of labels, For z m The second set of labels, For z m The nth group of labels.

[0064] Specifically, a range-based sharding method is proposed to shard the data elements with labels in the data space, including;

[0065]

[0066] Among them, low_1 is the lower limit of the first group range, up_1 is the upper limit of the first group range, low_2 is the lower limit of the second group range, up_2 is the upper limit of the second group range, low_q is the lower limit of the qth group range, up_q is the upper limit of the qth group range, For data elements under different scope constraints The labels belong to the range [low_1,up_1], [low_2,up_2], [low_q,up_q] respectively, and have:

[0067]

[0068] exist This is because the data elements have multiple sets of labels. In the range-based sharding method, the selection of data elements is blurred, and the attribute value of the label is more important. The mapping relationship between data elements and labels is one-to-many, and there are:

[0069]

[0070] Specifically, the multi-level caching step is as follows: multi-level cache is set at each node to accelerate data reading and writing operations, and it is proposed to use the Edge for Cloud model to cache data. On the basis of sharding the data elements with labels in the data space in a range-based sharding manner, q groups of subspaces are formed, each group of spaces contains independent labels, and the labels are cached through distributed edge nodes. Each label is regarded as a node, and the data space contains q groups of nodes for caching labels within the range, and the data space contains 1 cloud computing power, whose main function is to provide computing power and storage support for q edge nodes, which specifically includes the following:

[0071] In the framework of the q group subspace, there are labels: Note (1) ,note (2) ,…,note (q) are the labels in the first classification group, the labels in the second classification group, and the labels in the qth classification group respectively. The computing power set of the edge node is defined as computingpower=

[0072] {computingpower1,computingpower2,…,computingpower q}, among which, computingpower1, computingpower2,…,computingpower q They are the computing power of the first edge node, the computing power of the second edge node, and the computing power of the qth edge node. The computing power of the cloud computing power is defined as COMPUTINGPOWER CLOUD ,have

[0073] Specifically, in order to enhance the competitiveness of the data space, an overhead control mechanism is proposed to quantitatively control the cache of the data space. The overhead control mechanism includes:

[0074] (1) Edge node overhead and cloud overhead

[0075] The data attributes of edge nodes are defined as:

[0076] {Edge:computingpower,storagespace,offloading,cost edge ,cost cloud}

[0077] Among them, Edge is the edge node, computingpower is the computing power, storagespace is the storage space, offloading is the computing offloading, and the label will be offloaded to the specified edge node for caching. edge is the edge node overhead, cost cloud It is the cloud computing power expenditure;

[0078] Specifically, node overhead includes two situations: 1. When the node computing power or storage space supports the cache volume of the current computing offload, the node overhead belongs to the current edge node; 2. When the node computing power or storage space cannot meet the cache volume of the current computing offload, it is necessary to lease the edge node with idle node computing power or storage space in the data space for caching;

[0079] (2) Service selection for edge nodes

[0080] There are three ways to select services for edge nodes. If the computing power of the current edge node is sufficient but the storage space cannot provide services for the tag, the tag will wait online to complete the service; if the computing power of the current edge node is insufficient but the storage space can provide services for the tag, it can cooperate with the cloud computing power to participate in the cache service; if the computing power of the current edge node is insufficient and the storage space cannot provide services for the tag, it can call other edge nodes with computing power and storage space to collaborate on the cache task;

[0081] Specifically, for label online waiting, there are:

[0082]

[0083] C total ={1:Load1,2:Load2,…|EMPTY_ROOM}

[0084] Among them, ρ is the service intensity, μ is the service rate, λ is the unloading arrival rate, For the tag note *Unloading, computing power * is for the tag ntoe * The edge node of the service, Bandwidth is the network communication bandwidth, T cache is the cache time, is the total execution time when the tag is waiting online, C total is the total storage space of the edge node, Load1 is the first unloaded amount, Lodad2 is the second unloaded amount, and EMPTY_ROOM is the remaining storage space. but Enter the waiting service sequence of the edge node and complete the cache service in order;

[0085] Specifically, for collaborative cloud computing power to participate in cache services, there are:

[0086]

[0087] COST (2) =cost edge +cost cloud

[0088] in, is the total execution time of the collaborative cloud computing power participating in the cache service, The storage space for cloud computing power, COST (2) The total cost of coordinating cloud computing power to participate in caching services;

[0089] Specifically, other edge nodes with computing power and storage space coordinate caching services, including:

[0090]

[0091] COST (3) =cost edge +cost edge+

[0092] Among them, edge+ is the edge node that participates in the coordination of cache services, computingpower + For edge+ computing power, EMPTYROOM + The remaining space is edge+, The total execution time when coordinating cache services for other edge nodes with computing power and storage space, cost edge+ Cost of leasing edge+, COST (3) The total cost of coordinating caching services for other edge nodes with computing power and storage space;

[0093] (3) Parallel processing uses parallel processing technology to allow each node to independently process the sharded data in parallel, using multi-threading or multi-process technology to accelerate processing, using message queues to achieve communication and data synchronization between nodes, and processing different data fragments at the same time to shorten processing time;

[0094] (4) Consistency Check After data processing is completed, the data of each node is checked for consistency. The Raft protocol is used to perform consistency checks between nodes to ensure data consistency.

[0095] Furthermore, the heterogeneous data mapping mechanism dynamically generates heterogeneous data mapping rules according to the labels of data elements, including:

[0096] W=(zb)note T (note note T ) -1

[0097]

[0098] Where W is the weight matrix, z is the data element matrix, b is the bias vector, note is the label vector, z i is the i-th group of data elements, note j For z j Tags, b j For z j bias, use machine learning technology to learn from historical data and continuously optimize the mapping rules.

[0099] Furthermore, the data fusion includes: data extraction, data matching and data merging. Data extraction is to extract the data to be fused from various data sources; data matching is to match the extracted data to identify the same or similar data items; data merging is to merge the matched data items to generate a unified data view.

[0100] Beneficial effects of the present invention:

[0101] 1. The present invention adopts a logarithmic ratio transformation to pre-process the initial data elements. The traditional common logarithmic ratio transformation method is usually to compare the elements in a component vector with another selected reference element, and then take the natural logarithm of the ratio. This transformation will cause the data to lose its original proportional properties and can handle common scale problems and proportional dependence in proportional data. The logarithmic ratio transformation method provided by the present invention is to perform logarithmic ratio transformation on all components at the same time, but the reference is not a single component, but the geometric mean of all components. The advantage is that it keeps the dimension of the data unchanged and does not need to select a specific reference component. Therefore, it is more convenient and consistent when processing multivariate component data.

[0102] 2. The present invention proposes range-based data slicing, in which the data is characterized by a one-to-many mapping relationship. This means that in data slicing, the selection of data elements is non-independent, while the selection of labels is unique. In the range-based slicing method, the selection of data elements is blurred, and more emphasis is placed on the attribute value of the label, thereby enhancing the effect of data classification, being able to quickly retrieve the required data information within a specified range, and being able to extract data elements.

[0103] 3. The present invention proposes to cache data using the Edge for Cloud model. Based on the range-based sharding method for sharding the data elements with labels in the data space, q groups of subspaces are formed. Each group of spaces contains independent labels. The labels are cached through distributed edge nodes. An overhead control mechanism is proposed to quantitatively control the cache of the data space. In the service selection link of the edge node, the distributed data is processed. The situations of data queuing, insufficient computing power, and insufficient storage space are taken into consideration. A model is provided in which the labels are waiting online, the cloud computing power is coordinated to participate in the caching service, and other edge nodes with computing power and storage space coordinate the caching service. The overhead is used as a constraint to control the NP-hard problem within the range of feasible solution space.

[0104] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Distributed fusion method of multi-source heterogeneous data, including: Data preprocessing, data sharding, multi-level caching and distributed fusion.

2. The multi-source heterogeneous data distributed fusion method according to claim 1, wherein: The data preprocessing is to define a data space containing m groups of data elements, and to perform logarithmic ratio transformation on all components of the m groups of data elements simultaneously, but the benchmark is not a single component, but the geometric mean of all components, which is: Among them, x i is the i-th initial data element, z i is the i-th data element.

3. The multi-source heterogeneous data distributed fusion method according to claim 1, wherein: The data sharding step is as follows: define the data space as Z, the data space includes m data, each data includes n sets of labels, if the label does not contain any elements, the data space is not default, it is recorded as null, then there is , where z1 is the first group of data elements in data space Z, z2 is the second group of data elements in data space Z, and z m is the mth group of data elements in the data space Z, is the first set of labels for z1, is the second set of labels for z1, is the nth group label of z1, is the first set of labels for z2, is the second set of labels for z2, is the nth group label of z2, For z m The first set of labels, For z m The second set of labels, For z m The nth group of labels, a range-based sharding method is proposed to shard the data elements with labels in the data space, and there are; Among them, low_1 is the lower limit of the first group range, up_1 is the upper limit of the first group range, low_2 is the lower limit of the second group range, up_2 is the upper limit of the second group range, low_q is the lower limit of the qth group range, up_q is the upper limit of the qth group range, For data elements under different scope constraints The labels belong to the range [low_1,up_1], [low_2,up_2], [low_q,up_q] respectively, and have: Among them, Θ is an empty set, and there exists This is because the data elements have multiple sets of labels. In the range-based sharding method, the selection of data elements is blurred, and the attribute value of the label is more important. The mapping relationship between data elements and labels is one-to-many, and there are:

4. The multi-source heterogeneous data distributed fusion method according to claim 1, wherein: The multi-level caching step is specifically as follows: multi-level cache is set at each node to accelerate data reading and writing operations, and it is proposed to use the Edge for Cloud model to cache data. On the basis of sharding the data elements with labels in the data space based on the range-based sharding method, q groups of subspaces are formed, each group of spaces contains independent labels, and the labels are cached through distributed edge nodes. Each label is regarded as a node, and the data space contains q groups of nodes to cache the labels within the range, and the data space contains 1 cloud computing power, whose main function is to provide computing power and storage support for q edge nodes, which specifically includes the following: In the framework of the q group subspace, there are labels: Note (1) ,note (2) ,…,note (q) They are the labels of the first classification group, the labels of the second classification group, and the labels of the qth classification group respectively. The computing power set of the edge node is defined as computingpower={computingpower1,computingpower2,…,computingpower q }, among which, computingpower1, computingpower2,…,computingpower q They are the computing power of the first edge node, the computing power of the second edge node, and the computing power of the qth edge node. The computing power of the cloud computing power is defined as COMPUTINGPOWER CLOUD ,have 5. According to the multi-source heterogeneous data distributed fusion method of claim 4, in order to enhance the competitiveness of the data space, an overhead control mechanism is proposed to quantitatively control the cache of the data space. The overhead control mechanism includes: (1) Edge node overhead and cloud overhead The data attributes of edge nodes are defined as: {Edge:computingpower,storagespace,offloading,cost edge ,cost cloud } Among them, Edge is the edge node, computingpower is the computing power, storagespace is the storage space, offloading is the computing offloading, and the label will be offloaded to the specified edge node for caching. edge is the edge node overhead, cost cloud It is the cloud computing power overhead. The node overhead includes two situations:

1. When the node computing power or storage space supports the cache volume of the current computing offload, the node overhead belongs to the current edge node; 2. When the node computing power or storage space cannot meet the cache volume of the current computing offload, it is necessary to rent an edge node with idle node computing power or storage space in the data space for caching; (2) Service selection for edge nodes There are three ways to select services for edge nodes. If the computing power of the current edge node is sufficient but the storage space cannot provide services for the tag, the tag will wait online to complete the service; if the computing power of the current edge node is insufficient but the storage space can provide services for the tag, it can cooperate with the cloud computing power to participate in the cache service; if the computing power of the current edge node is insufficient and the storage space cannot provide services for the tag, it can call other edge nodes with computing power and storage space to collaborate on the cache task; For label online waiting, there are: C total ={1:Load1,2:Load2,…|EMPTY_ROOM} Among them, ρ is the service intensity, μ is the service rate, λ is the unloading arrival rate, For the tag note * Unloading, computing power * is for the tag note * The edge node of the service, Bandwidth is the network communication bandwidth, T cache is the cache time, is the total execution time when the tag is waiting online, C total is the total storage space of the edge node, Load1 is the first unloaded amount, Load2 is the second unloaded amount, and EMPTY_ROOM is the remaining storage space. but Enter the waiting service sequence of the edge node and complete the cache service in order; For collaborative cloud computing power to participate in cache services, there are: COST (2) =cost edge +cost cloud in, is the total execution time of the collaborative cloud computing power participating in the cache service, The storage space for cloud computing power, COST (2) The total cost of coordinating cloud computing power to participate in caching services; For other edge nodes with computing power and storage space, we coordinate caching services: COST (3) =cost edge +cost edge+ Among them, edge+ is the edge node that participates in the coordination of cache services, computingpower + For edge+ computing power, EMPTYROOM + The remaining space is edge+, The total execution time when coordinating cache services for other edge nodes with computing power and storage space, cost edge+ Cost of leasing edge+, COST (3) The total cost of coordinating caching services for other edge nodes with computing power and storage space; (3) Parallel processing uses parallel processing technology to allow each node to independently process the sharded data in parallel, using multi-threading or multi-process technology to accelerate processing, using message queues to achieve communication and data synchronization between nodes, and processing different data fragments at the same time to shorten processing time; (4) Consistency Check After data processing is completed, the data of each node is checked for consistency. The Raft protocol is used to perform consistency checks between nodes to ensure data consistency.

6. The multi-source heterogeneous data distributed fusion method according to claim 1, wherein: The heterogeneous data mapping mechanism dynamically generates heterogeneous data mapping rules according to the labels of data elements, including: W=(z-b)note T (note note T ) -1 Where W is the weight matrix, z is the data element matrix, b is the bias vector, note is the label vector, and z i is the i-th group of data elements, note j For z j Tags, b j For z j bias, use machine learning technology to learn from historical data and continuously optimize the mapping rules.

7. The multi-source heterogeneous data distributed fusion method according to claim 1, wherein: The data fusion includes: data extraction, data matching and data merging. Data extraction is to extract the data to be fused from various data sources; data matching is to match the extracted data to identify the same or similar data items; data merging is to merge the matched data items to generate a unified data view.

8. The multi-source heterogeneous data distributed fusion method according to claims 1-7, characterized in that: The methods provided in claims 1-7 are applied to multi-source heterogeneous data systems to improve the efficiency of large-scale data processing and the service quality of users.

Citation Information

Patent Citations

  • Constructive geochemical collaborative simulation hidden mine exploration risk evaluation method

    CN116384747A

  • Multi-source heterogeneous data fusion and mining method based on security space-time big data

    CN117992508A

  • Drug information storage method based on distributed edge calculation and multi-modal data

    CN118152481A

  • Industrial data analytics in a cloud platform

    US20140337429A1

  • Unstructured data fusion by content-aware concurrent data processing pipeline

    US20200082015A1

Cited By

  • Data processing system, method, device, medium and program product

    CN120743554A