A multi-source heterogeneous data fusion method and system

Abstract

The present application relates to the technical field of electric digital data processing, and more particularly to a multi-source heterogeneous data fusion method and system, which comprises connecting a distributed data acquisition platform to obtain a data set to be fused, constructing a hierarchical processing path, performing data format conversion and permission verification, and optimizing decision-making to generate a fusion strategy. By introducing an error model, calling a probability set, and a risk mapping mechanism, the accuracy, efficiency, and security of data fusion are improved. The present application can effectively solve the problems of low processing efficiency, insufficient flexibility, and limited data security protection for complex data structures, significantly improving the reliability and adaptability of data fusion.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method for fusing multi-source heterogeneous data, as well as a system for fusing multi-source heterogeneous data. Background Technology

[0002] With the continuous development of technology, multi-source heterogeneous data fusion technology has gradually become an important tool in various fields, including data processing and analysis. In the past, data fusion mainly relied on the simple integration of single-source or structured data. However, this method is inefficient when faced with complex data structures and is prone to inaccurate fusion results, failing to meet the modern demands for efficient and secure data fusion. Therefore, intelligent and adaptable multi-source heterogeneous data fusion methods and systems have gradually attracted attention and demonstrated good development potential.

[0003] Most existing multi-source heterogeneous data fusion methods are based on specific hardware architectures or scenario designs. Although they have certain advantages in terms of security or accuracy, their flexibility and scalability are limited. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for multi-source heterogeneous data fusion to solve the technical problems of insufficient efficiency, low flexibility, and limited data security in the processing of complex data structures in existing technologies, which leads to a decline in the accuracy and reliability of data fusion.

[0005] According to one aspect of the present invention, a method for fusing multi-source heterogeneous data is provided, the method being executed by a processor, comprising:

[0006] The resource management module connected to the distributed data acquisition platform calls the data classification unit to obtain the dataset to be fused, and collects the original data source information, target data attribute information and data quality assessment parameters of the dataset to be fused. The target data attribute information is information that characterizes the constraints that the data needs to meet during the fusion process.

[0007] Obtain the set of intermediate nodes of the dataset to be fused, and construct a hierarchical processing path using the original data source information, the target data attribute information, and the set of intermediate nodes;

[0008] The resource management module is connected to the adaptation unit, which performs data format conversion according to the target data attribute information and includes the data that conforms to the preset adaptation rules into the adaptation data set.

[0009] The resource management module is connected to the permission management unit. After associating the permission management unit with the adaptation unit, access permissions are verified to obtain the set of authorized users corresponding to each data in the adaptation data set.

[0010] Based on the adapted data set and the authorized user set, the hierarchical processing path is optimized, and the optimization decision result is output, wherein the optimization decision result is the processing path with the least data loss;

[0011] Based on the optimization decision results, a fusion strategy for the dataset to be fused is generated.

[0012] In some embodiments, the method further includes:

[0013] Connect to the historical data storage module and determine the initial planning path based on the optimal matching path between the original data source information and the target data attribute information;

[0014] On the initial planning path, a set of distributable nodes for the dataset to be fused is determined, wherein the set of distributable nodes consists of temporary storage nodes that satisfy the target data attribute information;

[0015] A first error model is introduced to identify the performance loss caused by data transmission delay for each node in all distributable node sets, and the identified node with the minimum loss is determined. The identified node is then used as the intermediate processing node of the dataset to be fused.

[0016] The initial planned path is segmented using the identified nodes to output a hierarchical processing path, wherein each layer of the hierarchical processing path has the smallest deviation from the initial planned path.

[0017] In some embodiments, the method further includes:

[0018] The adaptation unit classifies the data resources of the distributed data acquisition platform to obtain multiple types of data.

[0019] Collect basic information for each type of data, including data integrity information, data timeliness information, and data encryption status information, and generate a data status archive through a filtering mechanism;

[0020] The permission management unit collects basic information about users on the distributed data collection platform, including user activity information, user authentication level information, and user operation record information, and generates a user status archive through a filtering mechanism.

[0021] Based on the data status archive and the user status archive, the adaptation data set and the authorized user set are optimized.

[0022] In some embodiments, the method further includes:

[0023] Based on the adaptation data set and the data status archive, the probability of any data in the adaptation data set being called in the data status archive is identified, and a first call probability set corresponding to the adaptation data set is obtained.

[0024] Based on the authorized user set and the user status archive, the probability of any user in the authorized user set being invoked in the user status archive is identified, and a second invocation probability set corresponding to the authorized user set is obtained;

[0025] By weighting the first call probability set and the second call probability set, a comprehensive call probability set is output.

[0026] The optimization is performed on the comprehensive call probability set, and the optimization decision result is output.

[0027] In some embodiments, the method further includes:

[0028] Obtain the hierarchical processing path, wherein the hierarchical processing path includes a first processing layer, a second processing layer to an Nth processing layer, where N is the number of layers in the hierarchical processing path;

[0029] Obtain the comprehensive call probability set corresponding to each processing path, and obtain the corresponding identification data and identification user from the comprehensive call probability set corresponding to each processing path, as the hierarchical processing information of the dataset to be fused;

[0030] The fusion strategy for the dataset to be fused is automatically generated based on the hierarchical processing information.

[0031] In some embodiments, the data status archive and the user status archive are obtained by connecting to the scoring module. The data status archive is a database composed of information that the scores based on the basic data information reach the preset scoring standard, and the data status archive is a categorized database. The user status archive is a database composed of information that the scores based on the basic user information reach the preset scoring standard.

[0032] The scoring module includes an information parsing unit, which includes a decryption channel for parsing encrypted file information and a semantic analysis channel for parsing unstructured data.

[0033] In some embodiments, the method further includes:

[0034] By performing risk mapping on the comprehensive call probability set, a comprehensive risk probability set is output;

[0035] The comprehensive risk probability set is optimized to output a set of identified risk probabilities where the comprehensive risk probability is within a preset risk range.

[0036] The stability density of the identified risk probability set is optimized, and the optimized decision result is output.

[0037] According to another aspect of the present invention, a multi-source heterogeneous data fusion system is provided. The system includes a processor, and further includes a data acquisition unit, a path construction unit, an adaptation unit, an acquisition unit, an optimization decision-making unit, and a strategy generation unit, all data-connected to the processor; wherein...

[0038] The data acquisition unit is used to connect to the resource management module of the distributed data acquisition platform, call the data classification unit to obtain the dataset to be fused, and collect the original data source information, target data attribute information and data quality assessment parameters of the dataset to be fused. The target data attribute information is information that characterizes the constraints that the data needs to meet during the fusion process.

[0039] The path construction unit is used to obtain the set of intermediate nodes of the dataset to be fused, and to construct a hierarchical processing path using the original data source information, the target data attribute information, and the set of intermediate nodes;

[0040] The adaptation unit is used to connect the resource management module to call the adaptation unit, perform data format conversion according to the target data attribute information, and include data that conforms to the preset adaptation rules into the adaptation data set.

[0041] The acquisition unit is used to connect the resource management module to call the permission management unit, associate the permission management unit with the adaptation unit, perform access permission verification, and obtain the set of authorized users corresponding to each data in the adaptation data set.

[0042] The optimization decision unit is used to make optimization decisions on the hierarchical processing path based on the adaptation data set and the authorized user set, and output the optimization decision result, wherein the optimization decision result is the processing path with the least data loss;

[0043] The strategy generation unit is used to generate a fusion strategy for the dataset to be fused according to the optimization decision results.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This invention acquires the dataset to be fused through a distributed data acquisition platform, then constructs a hierarchical processing path, performs data format conversion and permission verification, optimizes the decision generation fusion strategy, and achieves efficient and accurate processing of multi-source data. By introducing an error model, calling probability sets, and risk mapping mechanisms, it improves the accuracy, efficiency, and security of data fusion. This invention can effectively solve the problem of improving the reliability and adaptability of data fusion. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the multi-source heterogeneous data fusion method of the present invention;

[0048] Figure 2 This is a schematic diagram of the multi-source heterogeneous data fusion system structure of the present invention. Detailed Implementation

[0049] The following will refer to the appendices in the embodiments of the present invention. Figures 1-2 The technical solutions in the embodiments of the present invention will be clearly and completely described together. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0050] Application Overview: In the current technological environment, when processing large-scale dynamic data streams, some technical solutions may affect the fusion efficiency due to insufficient support for real-time performance; other solutions have high requirements for data format and quality, resulting in a decrease in adaptability in real-world environments. These characteristics pose challenges to existing technologies when dealing with diverse data sources and complex application scenarios.

[0051] Example 1

[0052] Figure 1The flowchart of the multi-source heterogeneous data fusion method provided in the embodiments of the present invention is shown. The method is executed by a processor. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0053] The methods specifically include:

[0054] The resource management module, connected to the distributed data acquisition platform, calls the data classification unit to obtain the dataset to be fused. It collects the original data source information, target data attribute information, and data quality assessment parameters of the dataset to be fused. The target data attribute information represents the constraints that the data must meet during the fusion process. It then obtains the set of intermediate nodes for the dataset to be fused and constructs a hierarchical processing path based on the original data source information, target data attribute information, and the set of intermediate nodes. The resource management module calls the adaptation unit to convert the data format according to the target data attribute information and includes data that conforms to preset adaptation rules into the adaptation data set. The resource management module also calls the permission management unit to associate the permission management unit with the adaptation unit and perform access permission verification to obtain the set of authorized users corresponding to each data in the adaptation data set. Based on the adaptation data set and the set of authorized users, it makes optimization decisions on the hierarchical processing path and outputs the optimization decision result, which is the processing path with the least data loss. Finally, it generates a fusion strategy for the dataset to be fused according to the optimization decision result.

[0055] In some preferred embodiments, the method further includes: connecting a historical data storage module to determine an initial planning path based on the optimal matching path between the original data source information and the target data attribute information; determining a set of distributable nodes for the dataset to be fused on the initial planning path, wherein the set of distributable nodes is a set of temporary storage nodes that satisfy the target data attribute information; introducing a first error model to identify the performance loss caused by data transmission delay for each node in all distributable node sets, determining the identifier node with the minimum loss, and using the identifier node as an intermediate processing node for the dataset to be fused; segmenting the initial planning path with the identifier node, and outputting a hierarchical processing path, wherein each layer of the hierarchical processing path is the path with the minimum deviation from the initial planning path.

[0056] In some preferred embodiments, the method further includes: classifying the data resources of the distributed data acquisition platform according to the adaptation unit to obtain multiple types of data; collecting basic information of each type of data, including data integrity information, data timeliness information, and data encryption status information, and generating a data status archive through a filtering mechanism; collecting basic information of users of the distributed data acquisition platform according to the permission management unit, including user activity information, user authentication level information, and user operation record information, and generating a user status archive through a filtering mechanism; and optimizing the adaptation data set and the authorized user set based on the data status archive and the user status archive.

[0057] In some preferred embodiments, the method further includes: identifying the probability that any data in the adaptation data set will be invoked in the data status archive based on the adaptation data set and the data status archive, thereby obtaining a first invocation probability set corresponding to the adaptation data set; identifying the probability that any user in the authorized user set will be invoked in the user status archive based on the authorized user set and the user status archive, thereby obtaining a second invocation probability set corresponding to the authorized user set; outputting a comprehensive invocation probability set by weighting the first invocation probability set and the second invocation probability set; and optimizing the comprehensive invocation probability set to output an optimization decision result.

[0058] In some preferred embodiments, the method further includes: obtaining a hierarchical processing path, wherein the hierarchical processing path includes a first processing layer, a second processing layer to an Nth processing layer, wherein N is the number of layers in the hierarchical processing path; obtaining a comprehensive call probability set corresponding to each layer processing path, obtaining corresponding identification data and identification users from the comprehensive call probability set corresponding to each layer processing path, as hierarchical processing information of the dataset to be fused; and automatically generating a fusion strategy for the dataset to be fused according to the hierarchical processing information.

[0059] In some preferred embodiments, a data status archive and a user status archive are obtained by connecting to the scoring module. The data status archive is a database composed of information that has reached a preset scoring standard based on the data basic information, and the data status archive is a categorized database. The user status archive is a database composed of information that has reached a preset scoring standard based on the user basic information. The scoring module includes an information parsing unit, which includes a decryption channel for parsing encrypted file information and a semantic analysis channel for parsing unstructured data.

[0060] In some preferred embodiments, the method further includes: performing risk mapping on the comprehensive call probability set to output a comprehensive risk probability set; optimizing the comprehensive risk probability set to output an identified risk probability set whose comprehensive risk probability is within a preset risk range; and performing stability density optimization on the identified risk probability set to output an optimization decision result.

[0061] Example 2

[0062] Based on the same inventive concept as the multi-source heterogeneous data fusion method in Embodiment 1 above, such as Figure 2 As shown, the present invention also provides a multi-source heterogeneous data fusion system, the core of which lies in the synergistic effect of the resource management module of the distributed data acquisition platform and multiple functional units to achieve efficient processing, flexible adaptation and security assurance of complex data structures.

[0063] First, the data acquisition unit connects to the resource management module of the distributed data acquisition platform and calls the data classification unit to obtain the dataset to be fused. During this process, the data acquisition unit is responsible for collecting the original data source information, target data attribute information, and data quality assessment parameters of the dataset to be fused. The original data source information includes basic information such as the data generation device, timestamp, and transmission path; the target data attribute information characterizes the constraints that the data must meet during the fusion process, such as data format specifications and semantic consistency requirements; and the data quality assessment parameters cover indicators such as data integrity, timeliness, and encryption status. The collection of this information provides the necessary input basis for constructing the subsequent hierarchical processing path.

[0064] Next, the path construction unit obtains the set of intermediate nodes for the dataset to be fused based on the information output by the data acquisition unit, and constructs a hierarchical processing path based on the original data source information, target data attribute information, and the set of intermediate nodes. Specifically, the path construction unit first determines the initial planned path through the historical data storage module. This path is derived based on the optimal matching relationship between the original data source information and the target data attribute information. Subsequently, the distributable node set of the dataset to be fused is further determined on the initial planned path. These nodes refer to temporary storage nodes that satisfy the target data attribute information. To optimize path performance, a first error model is introduced to identify the performance loss caused by data transmission delay for each node in all distributable node sets, thereby determining the identifier node with the minimum loss as the intermediate processing node of the dataset to be fused. Finally, the initial planned path is segmented using the identifier node to output a hierarchical processing path, ensuring that the deviation of each layer path from the initial planned path is minimized.

[0065] After the path construction is completed, the adaptation unit connects to the resource management module, converts the data format according to the target data attribute information, and includes data that conforms to preset adaptation rules into the adaptation data set. The specific operations of the adaptation unit include classifying the data resources of the distributed data acquisition platform, acquiring multiple types of data and collecting basic information for each type, such as data integrity information, data timeliness information, and data encryption status information, and generating a data status archive through a filtering mechanism. Simultaneously, the permission management unit connects to the resource management module, associates the permission management unit with the adaptation unit, performs access permission verification, and obtains the authorized user set corresponding to each data in the adaptation data set. The permission management unit is also responsible for collecting basic information about users of the distributed data acquisition platform, including user activity information, user authentication level information, and user operation record information, and generating a user status archive through a filtering mechanism. The generation of the data status archive and user status archive provides important references for subsequent optimization of the adaptation data set and authorized user set.

[0066] In the optimization decision-making stage, the optimization decision-making unit optimizes the hierarchical processing path based on the adapted data set and the authorized user set, and outputs the optimization decision result. Specifically, the optimization decision-making unit first identifies the probability of any data in the adapted data set being called in the data status archive based on the adapted data set and the data status archive, obtaining a first call probability set; then, based on the authorized user set and the user status archive, it identifies the probability of any user in the authorized user set being called in the user status archive, obtaining a second call probability set. By weighting the first and second call probability sets, a comprehensive call probability set is output. Optimization is performed on the comprehensive call probability set, and the optimization decision result is output, which is the processing path with the least data loss. In addition, the optimization decision-making unit also performs risk mapping on the comprehensive call probability set to output a comprehensive risk probability set, and optimizes the comprehensive risk probability set to output an identified risk probability set whose comprehensive risk probability is within a preset risk range. Finally, stability density optimization is performed on the identified risk probability set, and the final optimization decision result is output.

[0067] Finally, the strategy generation unit automatically generates a fusion strategy for the datasets to be fused based on the optimization decision results output by the optimization decision unit. Specifically, the strategy generation unit first obtains the hierarchical processing path, which includes the first processing layer, the second processing layer, and so on up to the Nth processing layer, where N is the number of layers in the hierarchical processing path. Then, it obtains the comprehensive call probability set corresponding to each processing layer, and extracts the corresponding identifier data and identifier user from the comprehensive call probability set corresponding to each processing layer, as the hierarchical processing information for the datasets to be fused. Finally, it automatically generates a fusion strategy for the datasets to be fused according to the hierarchical processing information.

[0068] In practical applications, the multi-source heterogeneous data fusion system of this invention can be deployed in cross-regional enterprise data centers or cloud computing platforms to process heterogeneous data from different sources. For example, in the field of intelligent transportation, the system can collect data from multiple sensors, cameras, and vehicle terminals, perform format unification processing on the data through an adaptation unit, and ensure that sensitive data is only accessible to authorized users through an access control unit. The optimization decision-making unit dynamically adjusts the data processing path based on real-time traffic data and user needs, thereby achieving efficient data fusion and analysis. The scoring module plays a crucial role in this process; its information parsing unit parses encrypted file information through a decryption channel and parses unstructured data through a semantic analysis channel, ensuring the accuracy and reliability of the data status archive and the user status archive.

[0069] As can be seen from the above specific embodiments, the multi-source heterogeneous data fusion method and system of the present invention achieves efficient processing, flexible adaptation, and security assurance for complex data structures, solving the problems of insufficient processing efficiency, low flexibility, and limited data security assurance in the prior art. It has significant technical advantages and broad application prospects.

[0070] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principle of this invention will be further explained below in conjunction with a specific application scenario.

[0071] In the field of intelligent transportation, after deploying the multi-source heterogeneous data fusion system of this invention, the data acquisition unit first connects to multiple data source devices through the resource management module of the distributed data acquisition platform. For example, in an urban road monitoring scenario, the data acquisition unit acquires raw data from multiple sensors, cameras, and vehicle terminals. The data generated by these devices includes information such as vehicle speed, traffic light status, and road congestion. During the acquisition process, the data acquisition unit records the generator identifier, timestamp, and transmission path of each data point and stores this information as the raw data source information. Simultaneously, the data acquisition unit filters out suitable datasets to be fused based on the requirements of the target data attribute information, such as the data format needing to be JSON or XML, and semantic consistency requirements conforming to transportation standards and specifications. Furthermore, data quality assessment parameters such as data integrity, timeliness, and encryption status are also collected synchronously to ensure the reliability of subsequent processing.

[0072] Subsequently, the path construction unit begins its work based on the information provided by the data acquisition unit. For example, when it is necessary to fuse video data from different cameras with speed data from sensors, the path construction unit first retrieves the optimal matching path for similar scenarios through the historical data storage module. Assuming historical data shows that the optimal fusion path for video data and sensor data in a specific area is through intermediate nodes A and B, this path is determined as the initial planned path. Next, the path construction unit further analyzes the set of distributable nodes and identifies temporary storage nodes that can satisfy the target data attribute information. For example, nodes C and D are selected as candidate nodes due to their low latency and high bandwidth support. To optimize path performance, a first error model is introduced to calculate the performance loss of each node due to data transmission latency. After calculation, node C is determined as the identifier node, and the initial planned path is segmented based on this, ultimately outputting a hierarchical processing path. This process ensures that the deviation of each layer of the path from the initial planned path is minimized, thereby improving data transmission efficiency.

[0073] After constructing the hierarchical processing path, the adaptation unit begins data format conversion. For example, some sensor data may be in binary format, while camera data is in video stream format. The adaptation unit converts all data to JSON format according to the format specifications in the target data attribute information. During this process, the adaptation unit also collects basic information for each type of data, such as data integrity, timeliness, and encryption status, and generates a data status archive through a filtering mechanism. Simultaneously, the access control unit verifies user access permissions. For example, only users with advanced authentication levels from traffic management departments can access sensitive data, such as license plate numbers or driver personal information. The access control unit generates a user status archive by collecting user activity information, user authentication level information, and user operation records. These archives provide important references for subsequent optimization of the adaptation dataset and authorized user set.

[0074] In the optimization decision-making stage, the optimization decision-making unit optimizes the hierarchical processing path based on the adapted data set and the authorized user set. For example, when real-time traffic data shows a traffic accident on a certain road segment, the optimization decision-making unit prioritizes the rapid fusion of camera data and sensor data for that road segment. First, based on the adapted data set and the data status archive, the optimization decision-making unit identifies the probability of any data being accessed in the data status archive, obtaining a first access probability set. Then, based on the authorized user set and the user status archive, it identifies the probability of any user being accessed in the user status archive, obtaining a second access probability set. By weighting these two sets, a comprehensive access probability set is output. After optimization within the comprehensive access probability set, the optimization decision-making unit outputs the processing path with the minimum data loss. Furthermore, the optimization decision-making unit performs risk mapping on the comprehensive access probability set to output a comprehensive risk probability set, and optimizes this comprehensive risk probability set to output a set of identified risk probabilities where the comprehensive risk probability falls within a preset risk range. Finally, by optimizing the stability density of the identified risk probability set, the final optimization decision result is output.

[0075] Finally, the strategy generation unit automatically generates a fusion strategy for the dataset to be fused based on the optimization decision results output by the optimization decision unit. For example, for the traffic accident scenario mentioned above, the strategy generation unit first obtains a hierarchical processing path, which includes the first processing layer, the second processing layer, and so on up to the Nth processing layer. Subsequently, the strategy generation unit obtains the comprehensive call probability set corresponding to each processing layer and extracts the corresponding identifier data and identifier users from it as hierarchical processing information for the dataset to be fused. Finally, the fusion strategy for the dataset to be fused is automatically generated according to the hierarchical processing information. For example, the first processing layer is responsible for the initial integration of video data and sensor data, while the second processing layer performs semantic analysis on the integrated data to extract key information such as accident type and severity.

[0076] As can be seen from the above steps, the multi-source heterogeneous data fusion method and system of this invention achieve efficient processing, flexible adaptation, and security assurance for complex data structures in the field of intelligent transportation. For example, in traffic accident scenarios, the system can quickly fuse data from different sources to generate accurate accident reports, providing decision support for traffic management departments. Simultaneously, through strict control by the access control unit, sensitive data is ensured to be accessible only to authorized users, thereby enhancing data security. The scoring module plays a crucial role in this process; its information parsing unit parses encrypted file information through a decryption channel and parses unstructured data through a semantic analysis channel, ensuring the accuracy and reliability of the data status archive and user status archive. These characteristics give this invention significant technical advantages and broad application prospects in practical applications.

[0077] The specific example of the multi-source heterogeneous data fusion method in the aforementioned embodiment 1 is also applicable to the multi-source heterogeneous data fusion system of this embodiment. Through the foregoing description of the multi-source heterogeneous data fusion method, those skilled in the art can clearly understand the multi-source heterogeneous data fusion system of this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0079] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for fusing multi-source heterogeneous data, the method being executed by a processor, characterized in that, include: The resource management module connected to the distributed data acquisition platform calls the data classification unit to obtain the dataset to be fused, and collects the original data source information, target data attribute information and data quality assessment parameters of the dataset to be fused. The target data attribute information is information that characterizes the constraints that the data needs to meet during the fusion process. Obtain the set of intermediate nodes of the dataset to be fused, and construct a hierarchical processing path using the original data source information, the target data attribute information, and the set of intermediate nodes; The resource management module is connected to the adaptation unit, which performs data format conversion according to the target data attribute information and includes the data that conforms to the preset adaptation rules into the adaptation data set. The resource management module is connected to the permission management unit. After associating the permission management unit with the adaptation unit, access permissions are verified to obtain the set of authorized users corresponding to each data in the adaptation data set. Based on the adapted data set and the authorized user set, the hierarchical processing path is optimized, and the optimization decision result is output, wherein the optimization decision result is the processing path with the least data loss; Based on the optimization decision results, a fusion strategy for the dataset to be fused is generated.

2. The method according to claim 1, characterized in that, The method further includes: Connect to the historical data storage module and determine the initial planning path based on the optimal matching path between the original data source information and the target data attribute information; On the initial planning path, a set of distributable nodes for the dataset to be fused is determined, wherein the set of distributable nodes consists of temporary storage nodes that satisfy the target data attribute information; A first error model is introduced to identify the performance loss caused by data transmission delay for each node in all distributable node sets, and the identified node with the minimum loss is determined. The identified node is then used as the intermediate processing node of the dataset to be fused. The initial planned path is segmented using the identified nodes to output a hierarchical processing path, wherein each layer of the hierarchical processing path has the smallest deviation from the initial planned path.

3. The method according to claim 1, characterized in that, The method further includes: The adaptation unit classifies the data resources of the distributed data acquisition platform to obtain multiple types of data. Collect basic information for each type of data, including data integrity information, data timeliness information, and data encryption status information, and generate a data status archive through a filtering mechanism; The permission management unit collects basic information about users on the distributed data collection platform, including user activity information, user authentication level information, and user operation record information, and generates a user status archive through a filtering mechanism. Based on the data status archive and the user status archive, the adaptation data set and the authorized user set are optimized.

4. The method according to claim 3, characterized in that, The method further includes: Based on the adaptation data set and the data status archive, the probability of any data in the adaptation data set being called in the data status archive is identified, and a first call probability set corresponding to the adaptation data set is obtained. Based on the authorized user set and the user status archive, the probability of any user in the authorized user set being invoked in the user status archive is identified, and a second invocation probability set corresponding to the authorized user set is obtained; By weighting the first call probability set and the second call probability set, a comprehensive call probability set is output. The optimization is performed on the comprehensive call probability set, and the optimization decision result is output.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the hierarchical processing path, wherein the hierarchical processing path includes a first processing layer, a second processing layer to an Nth processing layer, where N is the number of layers in the hierarchical processing path; Obtain the comprehensive call probability set corresponding to each processing path, and obtain the corresponding identification data and identification user from the comprehensive call probability set corresponding to each processing path, as the hierarchical processing information of the dataset to be fused; The fusion strategy for the dataset to be fused is automatically generated based on the hierarchical processing information.

6. The method according to claim 3, characterized in that, The data status archive and the user status archive are obtained by connecting to the scoring module. The data status archive is a database composed of information that the scores based on the basic data information reach the preset scoring standards, and the data status archive is a categorized database. The user status archive is a database composed of information that the scores based on the basic user information reach the preset scoring standards. The scoring module includes an information parsing unit, which includes a decryption channel for parsing encrypted file information and a semantic analysis channel for parsing unstructured data.

7. The method according to claim 4, characterized in that, The method further includes: By performing risk mapping on the comprehensive call probability set, a comprehensive risk probability set is output; The comprehensive risk probability set is optimized to output a set of identified risk probabilities where the comprehensive risk probability is within a preset risk range. The stability density of the identified risk probability set is optimized, and the optimized decision result is output.

8. A multi-source heterogeneous data fusion system, the system comprising a processor, characterized in that, It also includes a data acquisition unit, a path construction unit, an adaptation unit, an acquisition unit, an optimization decision-making unit, and a strategy generation unit, all connected to the processor; wherein, The data acquisition unit is used to connect to the resource management module of the distributed data acquisition platform, call the data classification unit to obtain the dataset to be fused, and collect the original data source information, target data attribute information and data quality assessment parameters of the dataset to be fused. The target data attribute information is information that characterizes the constraints that the data needs to meet during the fusion process. The path construction unit is used to obtain the set of intermediate nodes of the dataset to be fused, and to construct a hierarchical processing path using the original data source information, the target data attribute information, and the set of intermediate nodes; The adaptation unit is used to connect the resource management module to call the adaptation unit, perform data format conversion according to the target data attribute information, and include data that conforms to the preset adaptation rules into the adaptation data set. The acquisition unit is used to connect the resource management module to call the permission management unit, associate the permission management unit with the adaptation unit, perform access permission verification, and obtain the set of authorized users corresponding to each data in the adaptation data set. The optimization decision unit is used to make optimization decisions on the hierarchical processing path based on the adaptation data set and the authorized user set, and output the optimization decision result, wherein the optimization decision result is the processing path with the least data loss; The strategy generation unit is used to generate a fusion strategy for the dataset to be fused according to the optimization decision results.

9. The system according to claim 8, characterized in that, The path construction unit is also used for: Connect to the historical data storage module and determine the initial planning path based on the optimal matching path between the original data source information and the target data attribute information; On the initial planning path, a set of distributable nodes for the dataset to be fused is determined, wherein the set of distributable nodes consists of temporary storage nodes that satisfy the target data attribute information; A first error model is introduced to identify the performance loss caused by data transmission delay for each node in all distributable node sets, and the identified node with the minimum loss is determined. The identified node is then used as the intermediate processing node of the dataset to be fused. The initial planned path is segmented using the identified nodes to output a hierarchical processing path, wherein each layer of the hierarchical processing path has the smallest deviation from the initial planned path.

10. The system according to claim 8, characterized in that, The optimization decision-making unit is also used for: Based on the adaptation data set and the data status archive, the probability of any data in the adaptation data set being called in the data status archive is identified, and a first call probability set corresponding to the adaptation data set is obtained. Based on the authorized user set and the user status archive, the probability of any user in the authorized user set being invoked in the user status archive is identified, and a second invocation probability set corresponding to the authorized user set is obtained; By weighting the first call probability set and the second call probability set, a comprehensive call probability set is output. The optimization is performed on the comprehensive call probability set, and the optimization decision result is output.

Images