Multi-source heterogeneous data integration system, method and equipment and storage medium

Abstract

The invention discloses a multi-source heterogeneous data integration system, method and device and a storage medium, and relates to the technical field of data processing, and the system comprises an information analysis module which is used for scanning each accessed data source through a target adapter so as to extract target information, analyzing the target information through a machine learning model, and obtaining an analysis result; the data source adaptation module is used for generating adaptation logic of each data source based on the analysis result and completing adaptation operation on the data sources according to the adaptation logic; the item set and rule determination module is used for cleaning and standardizing the data of each data source to obtain processed data, and determining a frequent item set and an association rule between the processed data based on an association rule mining algorithm; and the data association determination module is used for determining association of the processed data by utilizing the frequent item set, the association rule and the long-short-term memory network so as to analyze the processed data to complete data integration. According to the invention, intelligent integration of data is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-source heterogeneous data integration system, method, device and storage medium. Background Art

[0002] In the field of multi-source heterogeneous data integration, the current mainstream technologies mainly rely on data virtualization, service-oriented architecture (SOA, SOAframework), microservice architecture, and middleware technology. Data virtualization technology allows users to access data from different sources through a unified query interface without having to worry about the specific storage location and format of the data. Service-oriented architecture (SOA) achieves interoperability between different systems and promotes the integration of business processes by defining clear service interfaces. Microservice architecture decomposes applications into a set of small, independent services that can be deployed and expanded independently, improving the flexibility and maintainability of the system. Middleware technologies, such as Enterprise Service Bus (ESB, EnterpriseService Bus), provide functions such as data conversion, protocol conversion, and message routing to support communication and data exchange between different systems.

[0003] Although current mainstream technologies have made some progress in multi-source heterogeneous data integration, they still face some shortcomings in practical applications. First, existing data virtualization technologies may encounter performance bottlenecks when processing large-scale data sets, especially in scenarios with high data real-time requirements. Secondly, although SOA and microservice architectures improve the modularity and flexibility of the system, they also increase the complexity of the system, making management and maintenance more difficult. In addition, although middleware technology can realize communication between different systems, it often requires a lot of customized development to adapt to specific business needs and data formats, which not only increases development costs, but also prolongs the integration cycle. Therefore, how to improve data processing speed, enhance data security, and reduce operating costs in the data integration process is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a multi-source heterogeneous data integration system, method, device and storage medium, which can significantly reduce the tedious manual configuration and adaptation work in the traditional integration process, achieve rapid response to dynamic changes in data sources, thereby accelerating the deployment of integration projects and lowering technical barriers, improving data processing speed, enhancing data security, reducing operating costs, and improving user experience. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a multi-source heterogeneous data integration system, comprising:

[0006] An information analysis module, used to scan each connected data source using a target adapter to extract target information of each data source, and analyze the target information of each data source through a machine learning model built into the target adapter to obtain corresponding analysis results;

[0007] A data source adaptation module, used to generate adaptation logic corresponding to each of the data sources based on the analysis result through the target adapter, and complete the adaptation operation on the data source according to the adaptation logic;

[0008] An item set and rule determination module is used to clean and standardize the data corresponding to each of the data sources after the adaptation operation is completed, obtain the processed data, and determine the frequent item sets and association rules between each of the processed data based on the association rule mining algorithm;

[0009] The data association determination module is used to determine the association between each processed data using the frequent item set, the association rule and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data.

[0010] Optionally, the information analysis module includes:

[0011] A data source scanning unit is used to scan each connected data source using a target adapter to identify the type, structure and communication protocol of each data source, and extract the target information of each data source based on the natural language processing technology, the type, structure and communication protocol of the data source; the target information includes data format, update frequency and access interface.

[0012] Optionally, the data source adaptation module includes:

[0013] An adaptation logic generation unit is used to generate adaptation logic corresponding to each of the data sources through the target adapter based on the analysis results determined by using a support vector machine classifier; the adaptation logic includes any one or a combination of data extraction rules, data conversion rules, and data loading rules.

[0014] Optionally, the association includes any one or a combination of time series association, category association and numerical association.

[0015] Optionally, the system further includes:

[0016] A status evaluation module is used to collect target performance indicators of the data integration system in real time, and evaluate the current operating status of the data integration system according to the target performance indicators to obtain corresponding evaluation results; the target performance indicators include CPU usage, memory usage, data throughput, and data processing delay;

[0017] The resource configuration adjustment module is used to predict the resource demand and system load of the data integration system at a preset time based on the target performance indicators and the preset time series analysis model, and adjust the resource configuration of the data integration system according to the corresponding prediction results and the current operating status.

[0018] Optionally, the resource configuration adjustment module includes:

[0019] A resource configuration adjustment unit is used to increase or decrease the data processing threads of the data integration system, adjust the memory allocation of the data integration system, or optimize the data access path of the data integration system according to the corresponding prediction results and the current operating status.

[0020] Optionally, the system further includes:

[0021] The data encryption and access restriction module is used to encrypt data transmitted during the data integration process through quantum key distribution technology, restrict access to the corresponding encrypted data based on the user's role and permissions, and trigger an alarm when abnormal access behavior is detected.

[0022] In a second aspect, the present application discloses a multi-source heterogeneous data integration method, comprising:

[0023] Scanning each connected data source by using a target adapter to extract target information of each data source, and analyzing the target information of each data source by using a machine learning model built into the target adapter to obtain corresponding analysis results;

[0024] Generate, by the target adapter, adaptation logic corresponding to each of the data sources based on the analysis result, and complete the adaptation operation on the data source according to the adaptation logic;

[0025] After the adaptation operation is completed, the data corresponding to each of the data sources is cleaned and standardized to obtain processed data, and the frequent item sets and association rules between the processed data are determined based on an association rule mining algorithm;

[0026] The associations between the processed data are determined using the frequent item sets, the association rules, and the long short-term memory network, so that the processed data are analyzed based on the associations to complete the integration of multi-source heterogeneous data.

[0027] In a third aspect, the present application discloses an electronic device, comprising:

[0028] Memory, used to store computer programs;

[0029] A processor is used to execute the computer program to implement the multi-source heterogeneous data integration method as described above.

[0030] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the multi-source heterogeneous data integration method as described above is implemented.

[0031] The present application discloses a multi-source heterogeneous data integration system, including an information analysis module, which is used to scan each connected data source using a target adapter to extract the target information of each data source, and analyze the target information of each data source through a machine learning model built into the target adapter to obtain the corresponding analysis results; a data source adaptation module, which is used to generate the adaptation logic corresponding to each data source based on the analysis results through the target adapter, and complete the adaptation operation of the data source according to the adaptation logic; an item set and rule determination module, which is used to clean and standardize the data corresponding to each data source after the adaptation operation is completed, obtain the processed data, and determine the frequent item sets and association rules between each processed data based on the association rule mining algorithm; a data association determination module, which is used to determine the association between each processed data using the frequent item sets, the association rules and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data. It can be seen that the present application realizes rapid data adaptation through the adapter, and the adapter uses the machine learning model to dynamically detect and adapt the adaptation logic of different data sources to ensure seamless access to data. At the same time, deep learning algorithms are used to intelligently analyze and integrate data from different sources, explore the inherent connections between data, and achieve deep integration of data. In this way, the tedious manual configuration and adaptation work in the traditional integration process can be significantly reduced, and the dynamic changes of data sources can be quickly responded to, thereby speeding up the deployment of integration projects and lowering the technical threshold. It also improves the quality and efficiency of data processing, and deeply analyzes the inherent connections of data through machine learning models, providing enterprises with more accurate and comprehensive data views to support the making of complex decisions. Ultimately, it has a driving effect on enterprises, including increasing data processing speed, enhancing data security, reducing operating costs, improving user experience, and accelerating business decision-making processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0033] Figure 1 A schematic diagram of the structure of a multi-source heterogeneous data integration system disclosed in this application;

[0034] Figure 2 A flowchart of a multi-source heterogeneous data integration method disclosed in this application;

[0035] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] At present, existing data virtualization technologies may encounter performance bottlenecks when processing large-scale data sets, especially in scenarios with high data real-time requirements. Secondly, although SOA and microservice architectures have improved the modularity and flexibility of the system, they have also increased the complexity of the system, making management and maintenance more difficult. In addition, although middleware technology can achieve communication between different systems, it often requires a lot of customized development to adapt to specific business needs and data formats, which not only increases development costs, but also prolongs the integration cycle. In order to solve the above technical problems, the present application discloses a multi-source heterogeneous data integration system, which can significantly reduce the cumbersome manual configuration and adaptation work in the traditional integration process, and achieve rapid response to dynamic changes in data sources, thereby speeding up the deployment of integration projects and lowering technical barriers, improving data processing speed, enhancing data security, reducing operating costs, and improving user experience.

[0038] See also Figure 1 As shown, the embodiment of the present invention discloses a multi-source heterogeneous data integration system, including:

[0039] The information analysis module 11 is used to use the target adapter to scan the connected data sources so as to extract the target information of each data source, and analyze the target information of each data source through the machine learning model built into the target adapter to obtain corresponding analysis results.

[0040] In this embodiment, the first step in data source identification and adaptation is that the adapter first performs a comprehensive scan of the connected data source, and uses pattern recognition technology to identify the type, structure and communication protocol of the data source. Therefore, the information analysis module includes: a data source scanning unit, which is used to use the target adapter to scan each connected data source to identify the type, structure and communication protocol of each data source, and extract the target information of each data source based on the natural language processing technology, the type, structure and communication protocol of the data source; the target information includes data format, update frequency and access interface. Then, the adapter analyzes the characteristics of the data source, including data format, update frequency and access rights, through the built-in machine learning model. This process uses natural language processing (NLP) and machine learning technology to parse and understand the metadata descriptions of different data sources. First, the key information of the data source, such as data format, update frequency and access interface, is extracted using NLP technology.

[0041] The data source adaptation module 12 is used to generate adaptation logic corresponding to each of the data sources based on the analysis result through the target adapter, and complete the adaptation operation on the data source according to the adaptation logic.

[0042] In this embodiment, according to the analysis results, the adapter automatically generates adaptation logic, including rules for data extraction, conversion and loading. Therefore, the data source adaptation module of the present application includes: an adaptation logic generation unit, which is used to generate the adaptation logic corresponding to each of the data sources based on the analysis results determined by the support vector machine classifier through the target adapter; the adaptation logic includes any one or a combination of data extraction rules, data conversion rules and data loading rules. In this process, the adapter's self-learning mechanism is also included. By continuously monitoring the changes in the data source, the adapter can continuously optimize the adaptation strategy to adapt to the dynamic characteristics of the data source. In a specific embodiment, the adapter recognizes that the production equipment log is updated in time series, the supply chain management system provides real-time inventory data, and the sales records are stored in a database updated daily. The adapter generates corresponding adaptation logic for these data sources, such as extracting log data from the production equipment at regular intervals, obtaining inventory information from the supply chain system in real time, and loading sales records from the sales database every day. The present application matches the key information of the data source with the known data source type through the support vector machine (SVM) classifier to achieve automatic classification of the data source. Recurrent Neural Network (RNN) is used to identify and learn complex patterns and potential associations of data sources, thereby improving the accuracy and adaptability of pattern recognition. Finally, combined with reinforcement learning algorithms, the intelligent adapter can continuously learn and optimize in practical applications, automatically adjust the adaptation strategy to adapt to changes in data sources, and achieve highly adaptive data source identification and adaptation. This adaptive mechanism can automatically identify new data patterns and structural changes, and dynamically adjust data integration strategies, thereby significantly improving the accuracy and efficiency of data integration. And over time, its adaptive ability will continue to increase, allowing the system to show higher flexibility and robustness when dealing with complex and changing data environments.

[0043] The item set and rule determination module 13 is used to clean and standardize the data corresponding to each of the data sources after the adaptation operation is completed, obtain the processed data, and determine the frequent item sets and association rules between the processed data based on the association rule mining algorithm.

[0044] In this embodiment, after the data source adaptation is completed, the data fusion engine starts to perform deep data integration. First, the data is cleaned and standardized, invalid and duplicate data is removed, the data format of different data sources is unified, and the adapted data is analyzed in depth using machine learning algorithms to identify the correlation between data. The correlation here includes but is not limited to the correlation of time series, the correlation of categories, and the correlation of values. In this process, data preprocessing steps are first applied, including data cleaning, standardization, and normalization, to improve the accuracy of the analysis. Then, the Apriori algorithm is used to find the frequent item sets and association rules between data items, revealing the basic association pattern of data items.

[0045] The data association determination module 14 is used to determine the association between each processed data using the frequent item set, the association rule and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data.

[0046] In this embodiment, the potential patterns and trends between data are mined through algorithms, such as discovering the characteristics of customer groups through cluster analysis, or discovering the sales correlation between products through association rule learning. Then, these association rules and frequent item sets are input as features into the long short-term memory network (LSTM, long short-term memory network) to learn the deep representation of data and complex patterns in time series. Through training, the LSTM model can capture more subtle and nonlinear correlations between data, thereby identifying potential patterns and trends. The purpose of this step is to achieve deep fusion of data and provide a more comprehensive data view for subsequent data analysis and business decisions. For example, analyze the sales data of different products to identify which products have better sales performance in a specific region or time period; or analyze production data to find the relationship between equipment failure and production efficiency. Taking a new product line of an enterprise as an example, the data fusion engine integrates the sales data, production records and customer feedback of the product from all over the world. Through in-depth analysis, the engine found that a certain feature of the product was favored by consumers in the Asian market, but needed to be improved in the European market. At the same time, analysis of production data found that the failure rate of a certain production link was high, affecting the quality and production efficiency of the product. In this way, the application 2. uses deep learning models in data fusion, which can not only process the surface level integration of data, but also deeply analyze the complex relationships and potential patterns between data. It can extract richer information from multi-source heterogeneous data, achieve higher-level data understanding and knowledge discovery, provide more accurate data fusion results, and provide deeper insights for data analysis and decision support.

[0047] In addition, the present application is responsible for real-time monitoring of the running status of the data fusion engine through an adaptive optimizer. Therefore, the system of the present application also includes a status evaluation module, which is used to collect the target performance indicators of the data integration system in real time, and evaluate the current running status of the data integration system according to the target performance indicators to obtain corresponding evaluation results; the target performance indicators include CPU (Central Processing Unit) utilization rate, memory usage, data throughput and data processing delay; a resource configuration adjustment module, which is used to predict the resource requirements and system load of the data integration system at a preset time based on the target performance indicators and the preset time series analysis model, and adjust the resource configuration of the data integration system according to the corresponding prediction results and the current running status. The resource configuration adjustment module includes: a resource configuration adjustment unit, which is used to increase or decrease the data processing threads of the data integration system, adjust the memory allocation of the data integration system, or optimize the data access path of the data integration system according to the corresponding prediction results and the current running status.

[0048] That is to say, the present application evaluates the current state of the system by collecting key performance indicators such as CPU usage, memory usage, and data throughput. Based on these data, the optimizer uses an adaptive algorithm to optimize resource configuration and improve the overall performance of the system. The present application is based on an adaptive resource optimization algorithm based on real-time monitoring indicators, and adopts a multi-stage feedback control loop to dynamically adjust the configuration of computing resources. The algorithm first sets performance benchmarks and thresholds, and collects key performance indicators such as CPU usage, memory usage, I / O (Input / Output) throughput, and data processing delay in real time. Using these data, a time series analysis model is used to predict future resource requirements and system loads. Based on the prediction results, the algorithm dynamically adjusts resource allocation, such as automatically expanding computing nodes, adjusting memory allocation, or optimizing data flow processing paths. In addition, the algorithm also includes an optimization module that uses particle swarm optimization to find the optimal resource configuration to minimize costs and maximize system throughput. The goal is to improve the response speed and processing capacity of the system while ensuring the quality of data processing. In a specific embodiment, it is assumed that when processing data of a large-scale online trading system, the adaptive optimizer detects that the CPU usage of the system rises sharply during peak trading hours. The optimizer will automatically adjust resources, such as starting additional processing threads or reallocating memory resources, to ensure that transaction data can be processed and recorded quickly and accurately, thereby ensuring user experience and system stability. In this way, the system operation status is monitored in real time, and resource allocation and processing procedures are intelligently adjusted according to current data processing requirements and resource usage. When faced with data processing tasks of different loads and complexities, it can maintain efficient and stable operating performance. Compared with traditional manual optimization methods, it not only reduces the workload of system maintenance, but also improves the operating efficiency and reliability of the system.

[0049] It should be pointed out that the system of this application also includes: a data encryption and access restriction module, which is used to encrypt the data transmitted during the data integration process through quantum key distribution technology, restrict access to the corresponding encrypted data based on the user's role and authority, and trigger an alarm when abnormal access behavior is detected. Use advanced encryption technology to ensure the security of data during transmission and storage; provide a fine-grained access control mechanism to limit access to sensitive data based on the user's role and authority; provide data desensitization function so that it will not expose key data when sharing or displaying; continuously monitor potential security threats and trigger alarms when abnormal behavior is detected to prevent data from being unauthorized access or abuse. The encryption technology used in this application combines the stability of traditional encryption algorithms with the security of quantum computing, and adopts a dynamic key management system based on quantum key distribution (QKD) technology. The system uses the principles of quantum mechanics to generate keys that cannot be predicted or cloned, providing end-to-end encryption for data transmission. For example, when processing patient data in the healthcare industry, all personal identification information will be encrypted and ensure that only authorized medical personnel can access the health records of specific patients. At the same time, for research reports that need to be released publicly, the Guardian will desensitize the data, remove or replace all personally identifiable information to protect the privacy of patients.

[0050] In addition, the interactive window of the end user of the user interface of the present application supports a high degree of customization, allowing users to customize the interface layout and functional modules according to their business needs and operating habits. The user interface provides intuitive visualization tools that enable users to easily monitor the data integration process, configure data sources and view analysis results. It includes rich interactive elements, drag-and-drop data mapping, point-and-click model configuration and real-time data flow view. In a specific embodiment, in a supply chain management scenario, users may need to pay special attention to inventory levels and order fulfillment. Through the user interface, users can customize a dashboard to display real-time inventory data, order status and logistics information. Users can set threshold alerts. When the inventory level is below a certain value or the order processing is delayed, the system will automatically issue a notification to help users respond in time and optimize inventory management and order processing processes.

[0051] The present application discloses a multi-source heterogeneous data integration system, including an information analysis module, which is used to scan each connected data source using a target adapter to extract the target information of each data source, and analyze the target information of each data source through a machine learning model built into the target adapter to obtain the corresponding analysis results; a data source adaptation module, which is used to generate the adaptation logic corresponding to each data source based on the analysis results through the target adapter, and complete the adaptation operation of the data source according to the adaptation logic; an item set and rule determination module, which is used to clean and standardize the data corresponding to each data source after the adaptation operation is completed, obtain the processed data, and determine the frequent item sets and association rules between each processed data based on the association rule mining algorithm; a data association determination module, which is used to determine the association between each processed data using the frequent item sets, the association rules and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data. It can be seen that the present application realizes rapid data adaptation through the adapter, and the adapter uses the machine learning model to dynamically detect and adapt the adaptation logic of different data sources to ensure seamless access to data. At the same time, deep learning algorithms are used to intelligently analyze and integrate data from different sources, explore the inherent connections between data, and achieve deep integration of data. In this way, the tedious manual configuration and adaptation work in the traditional integration process can be significantly reduced, and the dynamic changes of data sources can be quickly responded to, thereby speeding up the deployment of integration projects and lowering the technical threshold. It also improves the quality and efficiency of data processing, and deeply analyzes the inherent connections of data through machine learning models, providing enterprises with more accurate and comprehensive data views to support the making of complex decisions. Ultimately, it has a driving effect on enterprises, including increasing data processing speed, enhancing data security, reducing operating costs, improving user experience, and accelerating business decision-making processes.

[0052] See also Figure 2 As shown, the embodiment of the present invention discloses a multi-source heterogeneous data integration method, including:

[0053] Step S11: Use the target adapter to scan the connected data sources to extract the target information of each data source, and analyze the target information of each data source through the machine learning model built into the target adapter to obtain corresponding analysis results.

[0054] Step S12: Generate, by the target adapter, adaptation logic corresponding to each of the data sources based on the analysis result, and complete the adaptation operation on the data source according to the adaptation logic.

[0055] Step S13: After the adaptation operation is completed, the data corresponding to each of the data sources is cleaned and standardized to obtain processed data, and the frequent item sets and association rules between the processed data are determined based on an association rule mining algorithm.

[0056] Step S14: Determine the association between each of the processed data using the frequent item sets, the association rules, and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data.

[0057] In summary, when integrating multi-source heterogeneous data, the present application first uses the target adapter to scan the connected data sources to extract the target information of each data source, and analyzes the target information of each data source through the machine learning model built into the target adapter to obtain the corresponding analysis results; then the target adapter generates the adaptation logic corresponding to each data source based on the analysis results, and completes the adaptation operation of the data source according to the adaptation logic; after the adaptation operation is completed, the data corresponding to each data source is cleaned and standardized to obtain the processed data, and the frequent item sets and association rules between each processed data are determined based on the association rule mining algorithm; finally, the association between each processed data is determined using the frequent item sets, the association rules and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data. It can be seen that the present application realizes rapid data adaptation through the adapter, and the adapter uses the machine learning model to dynamically detect and adapt the adaptation logic of different data sources to ensure seamless access to data. At the same time, a deep learning algorithm is used to intelligently analyze and integrate data from different sources, mine the intrinsic connection between data, and realize deep fusion of data. In this way, the tedious manual configuration and adaptation work in the traditional integration process can be significantly reduced, and the dynamic changes of data sources can be quickly responded to, thereby accelerating the deployment of integration projects and lowering the technical threshold. It also improves the quality and efficiency of data processing, and deeply analyzes the inherent connections of data through machine learning models, providing enterprises with more accurate and comprehensive data views to support the making of complex decisions. Ultimately, it has a driving effect on enterprises, including increasing data processing speed, enhancing data security, reducing operating costs, improving user experience and accelerating business decision-making processes.

[0058] Furthermore, the present application also discloses an electronic device. Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0059] Figure 3A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multi-source heterogeneous data integration method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0060] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0061] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0062] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the multi-source heterogeneous data integration method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0063] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the multi-source heterogeneous data integration method disclosed above is implemented. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0064] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0065] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0066] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0067] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0068] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-source heterogeneous data integration system, characterized in that: include: An information analysis module, used to scan each connected data source using a target adapter to extract target information of each data source, and analyze the target information of each data source through a machine learning model built into the target adapter to obtain corresponding analysis results; A data source adaptation module, used to generate adaptation logic corresponding to each of the data sources based on the analysis result through the target adapter, and complete the adaptation operation on the data source according to the adaptation logic; An item set and rule determination module is used to clean and standardize the data corresponding to each of the data sources after the adaptation operation is completed, obtain the processed data, and determine the frequent item sets and association rules between each of the processed data based on the association rule mining algorithm; The data association determination module is used to determine the association between each processed data using the frequent item set, the association rule and the long short-term memory network, so as to analyze the processed data based on the association to complete the integration of multi-source heterogeneous data.

2. The multi-source heterogeneous data integration system according to claim 1, characterized in that: The information analysis module comprises: A data source scanning unit is used to scan each connected data source using a target adapter to identify the type, structure and communication protocol of each data source, and extract the target information of each data source based on the natural language processing technology, the type, structure and communication protocol of the data source; the target information includes data format, update frequency and access interface.

3. The multi-source heterogeneous data integration system according to claim 1, characterized in that: The data source adaptation module includes: An adaptation logic generation unit is used to generate adaptation logic corresponding to each of the data sources through the target adapter based on the analysis results determined by using a support vector machine classifier; the adaptation logic includes any one or a combination of data extraction rules, data conversion rules, and data loading rules.

4. The multi-source heterogeneous data integration system according to claim 1, characterized in that: The association includes any one or a combination of time series association, category association and numerical value association.

5. The multi-source heterogeneous data integration system according to claim 1, characterized in that: Also includes: A status evaluation module is used to collect target performance indicators of the data integration system in real time, and evaluate the current operating status of the data integration system according to the target performance indicators to obtain corresponding evaluation results; the target performance indicators include CPU usage, memory usage, data throughput, and data processing delay; The resource configuration adjustment module is used to predict the resource demand and system load of the data integration system at a preset time based on the target performance indicators and the preset time series analysis model, and adjust the resource configuration of the data integration system according to the corresponding prediction results and the current operating status.

6. The multi-source heterogeneous data integration system according to claim 5, characterized in that: The resource configuration adjustment module includes: A resource configuration adjustment unit is used to increase or decrease the data processing threads of the data integration system, adjust the memory allocation of the data integration system, or optimize the data access path of the data integration system according to the corresponding prediction results and the current operating status.

7. The multi-source heterogeneous data integration system according to any one of claims 1 to 6, characterized in that: Also includes: The data encryption and access restriction module is used to encrypt data transmitted during the data integration process through quantum key distribution technology, restrict access to the corresponding encrypted data based on the user's role and permissions, and trigger an alarm when abnormal access behavior is detected.

8. A multi-source heterogeneous data integration method, characterized in that: include: Scanning each connected data source by using a target adapter to extract target information of each data source, and analyzing the target information of each data source by using a machine learning model built into the target adapter to obtain corresponding analysis results; Generate, by the target adapter, adaptation logic corresponding to each of the data sources based on the analysis result, and complete the adaptation operation on the data source according to the adaptation logic; After the adaptation operation is completed, the data corresponding to each of the data sources is cleaned and standardized to obtain processed data, and the frequent item sets and association rules between the processed data are determined based on an association rule mining algorithm; The associations between the processed data are determined using the frequent item sets, the association rules, and the long short-term memory network, so that the processed data are analyzed based on the associations to complete the integration of multi-source heterogeneous data.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the multi-source heterogeneous data integration method as claimed in claim 8.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the multi-source heterogeneous data integration method as claimed in claim 8 is implemented.

Images