Multi-source heterogeneous data management method and device, electronic equipment and storage medium

By processing and analyzing multi-source heterogeneous data in the partitioned data lake and building a target blockchain, the resource consumption and data association identification problems in multi-source heterogeneous data management are solved, and efficient data management and decision support are achieved.

CN118939738APending Publication Date: 2024-11-12CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411067130.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When processing multi-source heterogeneous data, the existing technology causes large infrastructure resources and bandwidth consumption, high data storage and management time cost, and the inability to effectively identify the multi-faceted associations and impacts of data, affecting data analysis and decision-making support.

Method used

Using a partitioned data lake-based method, multi-source heterogeneous data is obtained, metadata tagging, data processing and analysis are performed, analysis data is expanded, internal analysis data is generated and external analysis data is generated, and the target blockchain is built in the partitioned data lake.

Benefits of technology

It realizes convenient and efficient management of multi-source heterogeneous data, reduces infrastructure resources and bandwidth consumption, saves data storage and management time costs, improves the effectiveness of data analysis capabilities and decision-making support, and promotes data-driven decision-making and innovative development .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939738A_ABST
    Figure CN118939738A_ABST
Patent Text Reader

Abstract

The invention provides a multi-source heterogeneous data management method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of block chains and data processing. The data management method comprises the following steps: acquiring multi-source heterogeneous data of each device; processing and analyzing the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data; expanding the analysis data to obtain internal analysis data and external analysis data, and dumping the internal analysis data and the external analysis data into the partitioned data lake; and constructing a target block chain in the partitioned data lake. The problems that in the prior art, consumption of infrastructure resources and bandwidth is large, data storage and management time cost is high, multi-dimensional, dynamic and space-time related characteristics of data cannot be considered and presented, and follow-up data retrieval and decision making are inconvenient are at least solved. The method is suitable for data analysis processing scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of blockchain and data processing technology, and in particular to a method, device, electronic device and computer-readable storage medium for managing multi-source heterogeneous data. Background Art

[0002] With the rapid increase in the number of IoT devices, various sensors, smart devices and machines are interconnected. Modern enterprises and organizations integrate multiple data sources in various business scenarios, including internal systems, external data sources and data generated by the devices themselves. Device data usually comes from different manufacturers and uses different communication protocols and data formats. Therefore, the generation and use of multi-source heterogeneous data has become more common. Among them, internal systems such as ERP (Enterprise Resource Planning) and CRM (Customer Relationship Management) are more common.

[0003] At present, the data storage and management of equipment are usually based on database storage and management. However, due to the characteristics of multi-source heterogeneous data, such as large data volume, multiple data types, fast growth rate and the need for immediate processing, using databases to store and manage multi-source heterogeneous data will cause large consumption of infrastructure resources and bandwidth, high time cost of data storage and management, and the use of databases to store and manage multi-source heterogeneous data cannot consider and present the multi-dimensional, dynamic and time-space related characteristics of data, which is not convenient for subsequent data retrieval and decision-making. In addition, using databases to integrate multi-source heterogeneous data with inconsistent data types is not only time-consuming, but also may introduce errors or omissions. In critical tasks, it may lead to the inability to correctly analyze the operating status of the equipment or predict failures, causing enterprises to draw wrong conclusions during analysis, thereby affecting key decisions and the implementation of subsequent key tasks, and may even lead to uncontrollable accidents in equipment production. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and to provide a method, device, electronic device and computer-readable storage medium for managing multi-source heterogeneous data. The method can achieve convenient and efficient management of multi-source heterogeneous data, reduce the consumption of infrastructure resources and bandwidth, save the time cost of data storage and management, help identify the various correlations and impacts of data, improve insight and data analysis capabilities, enhance the effectiveness of decision support, and promote data-driven decision-making and innovative development in various industries.

[0005] In a first aspect, the present invention provides a method for managing multi-source heterogeneous data, comprising: acquiring multi-source heterogeneous data of each device; processing and analyzing the multi-source heterogeneous data based on a partitioned data lake to obtain analysis data; expanding the analysis data to obtain internal analysis data and external analysis data, and dumping the internal analysis data and the external analysis data into the partitioned data lake; and constructing a target blockchain in the partitioned data lake.

[0006] Preferably, the acquiring of multi-source heterogeneous data of each device specifically includes: collecting preparation data and operation data of the device from various data sources; and performing metadata tagging on the preparation data and the operation data, respectively, to obtain the multi-source heterogeneous data of each device.

[0007] Preferably, after acquiring the multi-source heterogeneous data of each device, and before processing and analyzing the multi-source heterogeneous data based on the partitioned data lake to obtain the analysis data, the multi-source heterogeneous data management method also includes: constructing a data lake based on the multi-source heterogeneous data, and uniformly defining the data lake to obtain a defined data lake; partitioning the defined data lake into an ingestion area, a preparation area, an analysis area, and a governance area to obtain a partitioned data lake.

[0008] Preferably, the analysis data includes extracted data, standardized format data, data reports and safety data reports. The processing and analysis of the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data specifically includes: extracting the multi-source heterogeneous data based on the ingestion area to obtain extracted data, and detecting changes in the multi-source heterogeneous data; converting the extracted data into a standardized format based on the preparation area to obtain standardized format data; analyzing the standard formatted data based on the analysis area and generating a data report; performing data governance on the extracted data, the standardized format data and the data report based on the governance area, and generating a safety data report.

[0009] Preferably, the analysis data is expanded to obtain internal analysis data and external analysis data, specifically including: based on the knowledge graph, formally conceptualizing the analysis data to obtain conceptualized analysis data; according to predefined fields, structuring the conceptualized analysis data to obtain preliminary structured analysis data; extracting semantic information of the preliminary structured analysis data; according to the semantic information, fusing and dividing the preliminary structured analysis data to obtain internal analysis data and external analysis data.

[0010] Preferably, the internal analysis data and the external analysis data include historical internal analysis data and external analysis data, as well as changed internal analysis data and external analysis data, and the construction of the target blockchain in the partitioned data lake specifically includes: generating a historical blockchain based on the historical internal analysis data and external analysis data; compressing the historical blockchain; filling and encapsulating the compressed historical blockchain according to the changed internal analysis data and external analysis data to obtain the target blockchain.

[0011] Preferably, the method of filling and encapsulating the compressed historical blockchain according to the changed internal analysis data and external analysis data to obtain the target blockchain specifically includes: collecting hash values ​​of the changed internal analysis data and external analysis data; timestamping the hash values ​​and sorting them; filling the changed internal analysis data and external analysis data into the compressed historical blockchain according to the sorting results; and encapsulating the filled historical blockchain to obtain the target blockchain.

[0012] In a second aspect, the present invention also provides a management device for multi-source heterogeneous data, including: an acquisition module, an analysis module, an extension module and a construction module, the acquisition module is used to acquire multi-source heterogeneous data of each device, the analysis module is connected to the acquisition module, and is used to process and analyze the multi-source heterogeneous data based on a partitioned data lake to obtain analysis data, the extension module is connected to the analysis module, and is used to expand the analysis data to obtain internal analysis data and external analysis data, and dump the internal analysis data and external analysis data into the partitioned data lake, and the construction module is connected to the extension module and is used to build a target blockchain in the partitioned data lake.

[0013] In a third aspect, the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the multi-source heterogeneous data management method provided in the first aspect.

[0014] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-source heterogeneous data management method provided in the first aspect is implemented.

[0015] The present invention provides a method, device, electronic device and computer-readable storage medium for managing multi-source heterogeneous data, which processes and analyzes the multi-source heterogeneous data of each device based on a partitioned data lake, obtains analysis data, expands analysis data, obtains internal analysis data and external analysis data, and builds a target blockchain in the partitioned data lake based on the internal analysis data and the external analysis data. Therefore, the present invention can achieve convenient and efficient management of multi-source heterogeneous data, reduce infrastructure resources and bandwidth consumption, save time costs for data storage and management, help identify multi-faceted data associations and impacts, improve insight and data analysis capabilities, enhance the effectiveness of decision support, and promote data-driven decision-making and innovative development in various industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for managing multi-source heterogeneous data according to Embodiment 1 of the present invention;

[0017] Figure 2 This is a flow chart of a data management method for industrial equipment according to Embodiment 2 of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of a data management device for industrial equipment according to Embodiment 3 of the present invention;

[0019] Figure 4 This is a schematic diagram of the structure of a computer device according to Embodiment 4 of the present invention;

[0020] Figure 5 This is a schematic diagram of the structure of a multi-source heterogeneous data management device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0022] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0023] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0024] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0025] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0026] It can be understood that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0027] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0028] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0029] Embodiment 1:

[0030] like Figure 1 As shown, this embodiment provides a method for managing multi-source heterogeneous data.

[0031] In this embodiment, with the development of Internet of Things technology and multi-system integration, multi-source heterogeneous data widely exists in various application scenarios, including industrial manufacturing, financial services, medical health, transportation, smart cities, scientific research and environmental protection. This embodiment takes industrial manufacturing as an example.

[0032] The management method of multi-source heterogeneous data includes:

[0033] S101, obtaining multi-source heterogeneous data from various devices.

[0034] Specifically, S101: acquiring multi-source heterogeneous data of each device, including steps S1011-S1012:

[0035] S1011, collecting equipment preparation data and operation data from various data sources.

[0036] In this embodiment, the equipment is taken as each intelligent device in the industrial park as an example. In the intelligent factory park, industrial equipment is all automated production equipment, which is usually controlled by programs. Generally, before the industrial equipment is produced, it is necessary to adjust the industrial equipment to a stable operating state. Therefore, the preparation data and operation data of each intelligent device in the industrial park are collected from various data sources, wherein the preparation data is the debugging data of the industrial equipment before production, the debugging data is used to adjust the industrial equipment to a stable operating state, and the operation data is the data of the industrial equipment when it is in a stable operating state. This embodiment can detect and analyze abnormal conditions in the operation of industrial equipment by acquiring the preparation data and operation data of industrial equipment for data storage and management, help to find problems in time and deal with them, and reduce production interruptions and losses.

[0037] S1012, metadata is marked on the preparation data and the operation data respectively to obtain multi-source heterogeneous data of each device.

[0038] In this embodiment, after collecting the preparation data and operation data of each smart device in the industrial park from various data sources, metadata tags are performed on the preparation data and the operation data to obtain multi-source heterogeneous data of each industrial device, wherein the metadata tags include but are not limited to: device identifier, data timestamp, data type, and data source. In this embodiment, by acquiring the preparation data and operation data of industrial equipment for data storage and management, it is possible to detect and analyze abnormal conditions in the operation of industrial equipment, help to discover problems in a timely manner and handle them, and reduce production interruptions and losses. By performing metadata tagging on the preparation data and the operation data, the meaning, source, and quality information of the preparation data and the operation data can be more clearly described and defined, the comprehensibility of the data can be improved, and different systems can better understand and use the data. It can also help to uniformly manage and integrate the preparation data and the operation data, promote data sharing and exchange between different data sources, and realize the integrated use and comprehensive analysis of data.

[0039] It should be noted that the multi-source heterogeneous data collected from various data sources can be divided into internal heterogeneous data and external heterogeneous data. Internal heterogeneous data refers to the data generated by the internal system of the factory park and each device itself, such as: equipment operating status, production indicators, fault alarms, production quality; external heterogeneous data refers to the data received or collected by the factory park from external data sources, such as: supply chain data, market demand data, weather data, and policy and regulatory change data. In addition, multi-source heterogeneous data can also be divided into unstructured data, semi-structured data and structured data according to the data type (i.e., data structure type). Among them, unstructured data refers to data without a specific format and organization, usually in text form, which is difficult to store and manage through traditional databases or tables, such as: equipment logs, sensor data streams, fault alarm texts; structured data refers to data with a clearly defined format and structure, which can be easily stored in a relational database and processed and analyzed through the SQL (Structured Query Language) query language, such as: equipment sensor data table, production schedule, production quality report; semi-structured data is between structured data and unstructured data, with a certain structure but not conforming to the specifications of traditional relational databases, such as: XML (ExtensibleMarkup Language, Extensible Markup Language), JSON (JavaScript Object Notation, a lightweight data representation method) format data, HTML (HyperText Markup Language, Hypertext Markup Language) web page text.

[0040] S102: Process and analyze multi-source heterogeneous data based on the partitioned data lake to obtain analysis data.

[0041] In this embodiment, data lake technology is a new technology and new architecture generated in the big data environment. Different from the data storage architecture of traditional databases, data lake supports data storage of all data types, including structured, semi-structured and unstructured data, and can retain original data, that is, data lake can store data in its original format until the data is needed. In this embodiment, data of different data types are brought together through the data lake, and data analysis is performed on data of different data types. Without pre-defined models, the latest data can be analyzed and mined immediately, thereby obtaining more immediate insights and conclusions. In addition, the big data architecture is scalable and can provide users with more real-time analysis.

[0042] Specifically, the analysis data includes extracted data, standardized format data, data reports and security data reports. Specifically, S102: Based on the partitioned data lake, multi-source heterogeneous data is processed and analyzed to obtain analysis data, including steps S1021-S1024:

[0043] In this embodiment, the analysis data is processed and analyzed based on multi-source heterogeneous data, without changing the data source of the multi-source heterogeneous data or the data structure type of the data. Therefore, by processing and analyzing the internal heterogeneous data and the external heterogeneous data based on the partitioned data lake, the analysis data of the internal heterogeneous data and the analysis data of the external heterogeneous data can be obtained; by processing and analyzing the structured data, semi-structured data and unstructured data in the multi-source heterogeneous data based on the analysis data lake, the analysis data of the structured data, the analysis data of the semi-structured data and the analysis data of the unstructured data can be obtained.

[0044] S1021, extracting multi-source heterogeneous data based on the ingestion area to obtain extracted data, and detecting changes in the multi-source heterogeneous data.

[0045] In this embodiment, with the help of metadata tags, multi-source heterogeneous data of all data types are ingested in their native formats without processing. The ingestion area is connected to each data source, and extracts multi-source heterogeneous data from each data source to obtain extracted data, and detects whether the multi-source heterogeneous data has changed. Among them, since the native formats of multi-source heterogeneous data of each data type are inconsistent, such as: the native format of structured data can be database tables, CSV (Comma-Separated Values, comma separated values) files and Excel files, the native format of semi-structured data can be XML, JSON and HTML, and the native format of unstructured data can be txt, img, therefore, multiple formats of extracted data can be obtained, that is, the extracted data includes structured data in database tables, CSV, Excel file formats, semi-structured data in XML, JSON and HTML formats, and unstructured data in txt, img formats. The ingestion process can be implemented by stream processing, batch processing, real-time processing or hybrid processing technology.

[0046] S1022, converting the extracted data into a standardized format based on the preparation area to obtain standardized format data.

[0047] In this embodiment, data processing is an inevitable step in data analysis. In the preparation area, multi-source heterogeneous data extracted in the native format, that is, the extracted data, can be converted into standardized format data according to preset requirements, that is, multi-source heterogeneous data in different formats are converted into the same format, such as CSV, JSON, and txt are converted into a unified format and data type, such as: standardized table data. The data processing process in the preparation area can also be implemented through batch processing and real-time processing technology, that is, batch processing and real-time processing of the extracted data are performed through batch processing and real-time processing technology, so that a quick response can be made and bandwidth occupancy can be effectively reduced.

[0048] S1023, analyzing the standard formatted data based on the analysis area and generating a data report.

[0049] In this embodiment, in the analysis area, the standardized format data is effectively read and processed by the analysis tool for different data analyses, including but not limited to: reports, statistical analysis, and machine learning algorithms. After the data analysis, a data report is generated, wherein the data report includes but is not limited to: a statistical analysis report and a data intelligence analysis report.

[0050] S1024, perform data governance on extracted data, standardized format data, and data reports based on the governance area, and generate a security data report.

[0051] In this embodiment, data governance is applied to all areas of the partitioned data lake, responsible for ensuring data security, data quality, data lifecycle, data access, and metadata management of the partitioned data lake, and preventing the partitioned data lake from deteriorating into a data swamp, where a data swamp refers to a large-scale data repository that is completely inaccessible to users. In other words, the governance zone is responsible for data governance of the data collected or generated by the ingestion zone, preparation zone, analysis zone, and governance zone of the partitioned data lake.

[0052] Optionally, after S101: acquiring multi-source heterogeneous data from each device, and before S102: processing and analyzing the multi-source heterogeneous data based on the partitioned data lake to obtain the analysis data, the multi-source heterogeneous data management method also includes: constructing a data lake based on the multi-source heterogeneous data, and uniformly defining the data lake to obtain a defined data lake; partitioning the defined data lake into an ingestion area, a preparation area, an analysis area, and a governance area to obtain a partitioned data lake.

[0053] In this embodiment, a data lake is constructed according to the preparation data and operation data of industrial equipment, and the data lake is uniformly defined to obtain a defined data lake, wherein the uniform definition includes but is not limited to: defining the target industrial equipment type. The defined data lake is partitioned into an ingestion area, a preparation area, an analysis area, and a governance area according to the function to obtain a partitioned data lake, wherein the ingestion area, the preparation area, the analysis area, and the governance area each contain two layers, namely, a data processing layer and an output storage layer, so that the partitioned data lake can normally access, process and analyze multi-source heterogeneous data, and store the processing and analysis results of multi-source heterogeneous data in the ingestion area, the preparation area, the analysis area, and the governance area in their respective output storage layers. In this embodiment, by partitioning the data lake into an ingestion area, a preparation area, an analysis area, and a governance area, the data lake can be transformed from a single-structure architecture to a multi-structure architecture, thereby realizing a more orderly data processing and analysis process. Different areas of the data lake (i.e., the ingestion area, the preparation area, the analysis area, and the governance area) have different functions and purposes, and each undertakes different tasks and responsibilities, thereby providing a more flexible and controllable environment for data analysis and processing. You can also define access rights for each output storage layer in the ingestion area, preparation area, analysis area, and governance area. For example, the data in the output storage layer of the ingestion area, preparation area, and analysis area can be accessed and called not only by the extension module, but also by the governance area. Therefore, while the orderly data processing and analysis process of the partitioned data lake can be realized, the processing and analysis results generated by each data processing and analysis step can be stored in an orderly manner, which is convenient for the subsequent calling of the processing and analysis results of the ingestion area, preparation area, analysis area, and governance area, that is, analyzing the data for extended analysis.

[0054] It should be noted that the output storage layers of the ingestion area, preparation area, analysis area and governance area can periodically clear their respective processing and analysis results according to the clearing strategy. For example, after expanding the analysis data to obtain internal analysis data and external analysis data, and dumping the internal analysis data and external analysis data into the partitioned data lake, the analysis data in the output storage layers of the ingestion area, preparation area, analysis area and governance area are cleared.

[0055] S103, expanding the analysis data to obtain internal analysis data and external analysis data, and dumping the internal analysis data and the external analysis data into the partitioned data lake.

[0056] In this embodiment, compared with machine learning and deep learning, knowledge graphs do not require a large amount of training data, providing greater potential for solving problems in many fields. This embodiment expands the analytical data of the partitioned data lake based on the knowledge graph, improves the discoverability and reusability of the analytical data, and establishes a new form of data analysis that can capture semantically interconnected large-scale data sets.

[0057] In addition, similarly, the data lake is partitioned into the ingestion area, preparation area, analysis area, and governance area according to the function to obtain the partitioned data lake. The partitioned data lake can also include a storage area, which is used to store the dumped internal analysis data and external analysis data. In essence, internal analysis data refers to the data obtained after the analysis data of internal heterogeneous data is expanded, and external analysis data refers to the data obtained after the analysis data of external heterogeneous data is expanded.

[0058] It should be noted that the present embodiment can also call the internal analysis data and external analysis data of the storage area through the application layer API (Application Programming Interface) to perform advanced data analysis, and the advanced data analysis includes but is not limited to: predictive analysis, intelligent analysis. Specifically, the analysis data is expanded to obtain internal analysis data and external analysis data, including: based on the knowledge graph, formally conceptualizing the analysis data to obtain conceptualized analysis data; according to the pre-defined field, structuring the conceptualized analysis data to obtain preliminary structured analysis data; extracting the semantic information of the preliminary structured analysis data; according to the semantic information fusion and division of the preliminary structured analysis data, to obtain internal analysis data and external analysis data.

[0059] In this embodiment, the analysis data of the partitioned data lake based on the knowledge graph expansion is specifically as follows: ① Formal conceptualization indicates the logical design of the knowledge graph, and through conceptual analysis of the entities, relationships and attributes in the knowledge graph, explores the hierarchical structure and semantic relationship between concepts, identifies relevant concepts and attributes in the analysis data, and generates conceptualized analysis data. ② According to the pre-defined domain ontology description, that is, the formal representation of the domain knowledge structure, the concepts, entities and relationships of the domain of interest are captured, and the pre-defined domain ontology structure is combined with the concepts, entities and relationships of the domain of interest to obtain preliminary structured analysis data. Since the analysis data includes extracted data, standardized format data, data reports and data security reports, the analysis data is essentially heterogeneous data. Through ① and ②, the heterogeneous analysis data is converted into consistent graph structure data. ③ Using natural language processing and machine learning technology, semantic information is extracted from the preliminary structured analysis data. The semantic information includes but is not limited to context information and time series information. ④ According to the semantic information association and integration of preliminary structured analysis data, including but not limited to: entity relationship, attribute value association, generate semantic analysis data with more depth and breadth, that is, the fused preliminary structured analysis data; and according to the data source identification of the fused preliminary structured data itself and the extracted semantic information, determine the data source type of the fused preliminary structured analysis data, and divide the fused preliminary structured analysis data into internal analysis data and external analysis data according to the data source type, wherein the data source type includes internal data source and external data source. This embodiment converts analysis data into preliminary structured analysis data based on knowledge graph and pre-defined domains to solve specific subject domain knowledge problems, which can improve the efficiency and accuracy of subsequent data analysis and mining; by extracting the semantic information of preliminary structured analysis data, and fusing and dividing preliminary structured analysis data according to semantic information, generating internal analysis data and external analysis data with more depth and breadth, it is possible to have a deeper understanding of the meaning and association of preliminary structured analysis data, and enhance the effect of data mining and data integration.

[0060] S104, building a target blockchain in the partitioned data lake.

[0061] In this embodiment, blockchain refers to a chain of several blocks, each of which stores certain information. Each block is connected into a chain in the order of their generation, i.e., blockchain. The blockchain is stored in all servers. As long as there is one server in the entire system that can work, the entire blockchain is safe. The server is called a blockchain management node in the blockchain system, which provides storage space and computing power support for the entire blockchain system. If you want to modify the information in the blockchain, you must obtain the consent of more than half of the blockchain management nodes and modify the information in all blockchain management nodes. The blockchain management nodes are usually in the hands of different entities, so it is extremely difficult to tamper with the information in the blockchain. In this embodiment, the blockchain management node is deployed in the partitioned data lake.

[0062] Compared with traditional storage networks, blockchain has two core characteristics: one is that data is difficult to tamper with, and the other is decentralization. Based on the two core characteristics, the information recorded by the blockchain is more authentic and reliable, which can help solve the problem of people not trusting each other. This embodiment can enhance data security, improve data traceability, realize data sharing and access control, apply smart contracts, and realize decentralized data management by building a blockchain in a partitioned data lake to store internal and external analysis data of each industrial device, which helps to improve the efficiency, security and credibility of industrial data management and analysis and promote the development of industrial intelligence.

[0063] Specifically, the internal analysis data and the external analysis data include historical internal analysis data and external analysis data, and changed internal analysis data and external analysis data.

[0064] Specifically, S104: constructing a target blockchain in the partitioned data lake, including steps S1041-S1043:

[0065] S1041, generate a historical blockchain based on historical internal analysis data and external analysis data.

[0066] In this embodiment, if the ingestion area of ​​the partitioned data lake is the first time that a change in multi-source heterogeneous data is detected after the multi-source heterogeneous data of each industrial device is initially accessed, then the historical internal analysis data and external analysis data are the initial internal analysis data and external analysis data obtained after the initial access multi-source heterogeneous data is processed and expanded by the partitioned data lake. The historical blockchain is the initial blockchain constructed by generating and connecting several blocks based on the initial internal analysis data and external analysis data.

[0067] It should be noted that if it is not the first time that the ingestion zone detects changes in multi-source heterogeneous data, then the process of generating a historical blockchain based on historical internal analysis data and external analysis data is equivalent to steps S1041-S1043. That is, the last time a change in multi-source heterogeneous data was detected, the target blockchain generated by executing steps S1041-S1043 was the historical blockchain.

[0068] S1042, compress historical blockchain.

[0069] In this embodiment, before generating and compressing the historical blockchain, it also includes deploying a chain compression module and a block production module in the partitioned data lake. The chain compression module and the block production module are the main functional modules of the blockchain management node. Among them, the block production module is used to generate a number of blocks based on internal analysis data and external analysis data, and merge a number of blocks to build a blockchain. The chain compression module is used to compress the data in the blockchain. This embodiment constructs and compresses the blockchain through the chain compression module and the block production module, which can achieve rapid storage of internal analysis data and external analysis data, reduce storage space and improve reading performance.

[0070] It should be noted that this embodiment can selectively compress the historical blockchain, that is, after receiving the compression request, the chain compression module responds to the compression request to start the compression algorithm; after the compression is completed, the chain compression module will also output the compression result.

[0071] S1043, fill and encapsulate the compressed historical blockchain according to the changed internal analysis data and external analysis data to obtain the target blockchain.

[0072] Specifically, S1043: filling and encapsulating the compressed historical blockchain according to the changed internal analysis data and external analysis data to obtain the target blockchain, including: collecting hash values ​​of the changed internal analysis data and external analysis data; timestamping the hash values ​​and sorting them; filling the changed internal analysis data and external analysis data into the compressed historical blockchain according to the sorting results; encapsulating the filled historical blockchain to obtain the target blockchain.

[0073] In this embodiment, hash values ​​of the changed internal analysis data and external analysis data are collected; the hash values ​​of the changed internal analysis data and external analysis data are timestamped and sorted; and then, based on the sorting results, the changed internal analysis data and external analysis data are filled into the compressed historical blockchain.

[0074] Due to the characteristics of blockchain, each block contains the hash value of the previous block. By adding the hash value of the previous block to the current block, the blocks can be connected together to form a blockchain, ensuring the immutability of the blockchain. Once an attempt is made to modify the data in the blockchain, the consistency of the hash value will be destroyed, rendering the entire blockchain invalid. Therefore, when a single point failure occurs, it is necessary to access all other blockchain management nodes to recover.

[0075] In this embodiment, the hash values ​​of the changed internal analysis data and external analysis data are used as the hash values ​​of the auxiliary information of the t-th block, and the hash values ​​of each block of the target blockchain are calculated according to Ht=h(ψt) and ψt=(Ht-1,h′(Bt)), where Bt represents the t-th block, h′(Bt) represents the hash value of the auxiliary information of the t-th block, Ht-1 represents the hash value of the t-1-th block, Ht represents the hash value of the t-th block, and h and h′ are two hash functions. The hash values ​​of each block of the target blockchain are used as the header data of each block, and the filled historical blockchain is encapsulated to obtain the target blockchain. This embodiment calculates the hash value of the current block based on the hash values ​​of the changed internal analysis data and external analysis data, as well as the hash value of the previous block, and uses the hash value of the current block as the block header to connect each block, generate and encapsulate the target blockchain, and complete the entire process of blockchain generation of multi-source heterogeneous data of each industrial equipment. After the blockchain management node verifies the block, the data of the block can be modified, and the multi-source heterogeneous data of each industrial equipment can be controlled in time to avoid uncontrollable production problems.

[0076] It needs to be explained that when the hash value of each block of the target blockchain is used as the header data of each block, the filled historical blockchain is encapsulated, and the target blockchain is obtained, the hash value of each block of the target blockchain can be placed in front of the compression depth to facilitate the compression of subsequent blockchains.

[0077] Node (i.e. blockchain management node) management can also be performed in the target blockchain, including but not limited to: node information management and joining, exiting and dormancy of network virtual nodes in the target blockchain. In this embodiment, a node management module can be deployed, which provides calls for joining and exiting operations for blockchain management nodes; the node management module can also be used to manage the identity information of blockchain management nodes, the production rounds and health of blockchain management nodes; in addition, the node management module can also provide blockchain management node information to other blockchain management nodes through an interface.

[0078] The present embodiment provides a method for managing multi-source heterogeneous data, which processes and analyzes the multi-source heterogeneous data of each device in the partitioned data lake, obtains analysis data, expands the analysis data, obtains internal analysis data and external analysis data, and builds a target blockchain in the partitioned data lake based on the internal analysis data and the external analysis data, so as to realize convenient and efficient management of multi-source heterogeneous data, reduce infrastructure resources and bandwidth consumption, save the time cost of data storage and management, help identify the multi-faceted associations and influences of data, improve the ability of insight and data analysis, enhance the effectiveness of decision support, and promote data-driven decision-making and innovative development in various industries. In addition, by acquiring the preparation data and operation data of industrial equipment for data storage and management, it is possible to detect and analyze abnormal conditions in the operation of industrial equipment, help to find problems in time and deal with them, and reduce production interruptions and losses. By metadata tagging the preparation data and operation data, the meaning, source, and quality information of the preparation data and operation data can be more clearly described and defined, the comprehensibility of the data can be improved, and different systems can better understand and use the data. It can also help to uniformly manage and integrate the preparation data and operation data, promote data sharing and exchange between different data sources, and realize the integrated use and comprehensive analysis of data. The data lake brings together data of different data types and performs data analysis on them. Without pre-defined models, the latest data can be analyzed and mined immediately, thereby obtaining more immediate insights and conclusions. The big data architecture is scalable and can provide users with more real-time analysis. By partitioning the data lake into ingestion, preparation, analysis, and governance areas, the data lake can be transformed from a single-structure architecture to a multi-structure architecture, achieving a more orderly data processing and analysis process. The functions and purposes of different areas of the data lake (i.e., ingestion, preparation, analysis, and governance) are different, and each has different tasks and responsibilities, thus providing a more flexible and controllable environment for data analysis and processing. The analytical data of the partitioned data lake based on the knowledge graph is expanded to improve the discoverability and reusability of analytical data, and establish a new form of data analysis that can capture semantically interconnected large-scale data sets. Based on knowledge graphs and pre-defined domains, analytical data is converted into preliminary structured analytical data to solve specific subject area knowledge problems, which can improve the efficiency and accuracy of subsequent data analysis and mining; by extracting and integrating the semantic information of preliminary structured analytical data, deeper and broader internal analytical data and more comprehensive external analytical data are generated, which can more deeply understand the meaning and association of preliminary structured analytical data and enhance the effect of data mining and data integration.By building a blockchain in a partitioned data lake to store the internal and external analysis data of each industrial device, data security can be enhanced, data traceability can be improved, data sharing and access control can be achieved, smart contracts can be applied, and decentralized data management can be achieved, which will help improve the efficiency, security and credibility of industrial data management and analysis, and promote the development of industrial intelligence. By building and compressing the blockchain through the chain compression module and the block production module, the internal and external analysis data can be quickly stored, and the storage space can be reduced and the reading performance can be improved. Based on the hash values ​​of the changed internal and external analysis data, as well as the hash value of the previous block, the hash value of the current block is calculated, and the hash value of the current block is used as the block header to connect each block, generate and encapsulate the target blockchain, and complete the entire process of blockchain generation of multi-source heterogeneous data of each industrial device. After the blockchain management node verifies the block, the data of the block can be modified, which can timely control the multi-source heterogeneous data of each industrial device to avoid uncontrollable production problems.

[0079] Embodiment 2:

[0080] like Figure 2 As shown, this embodiment provides a data management method for industrial equipment. The data management method for industrial equipment includes:

[0081] S1, collects multi-source heterogeneous data from various industrial equipment.

[0082] In this embodiment, the multi-source heterogeneous data of each industrial device is Figure 2 The data of each industrial equipment in the industrial park. Taking the industrial equipment as the various intelligent devices in the industrial park as an example, in the intelligent factory park, the industrial equipment is all automated production equipment, which is usually controlled by programs. Generally, before the industrial equipment is produced, it is necessary to adjust the industrial equipment to a stable operation state. Therefore, the preparation data and operation data of each intelligent device in the industrial park are collected from various data sources, among which the preparation data is the debugging data of the industrial equipment before production, the debugging data is used to adjust the industrial equipment to a stable operation state, and the operation data is the data of the industrial equipment when it is in a stable operation state.

[0083] After collecting the preparation data and operation data of each smart device in the industrial park from various data sources, metadata tags are performed on the preparation data and the operation data to obtain multi-source heterogeneous data of each industrial device, where the metadata tags include but are not limited to: device identifier, data timestamp, data type, and data source.

[0084] It should be noted that the multi-source heterogeneous data collected from various data sources can be divided into internal heterogeneous data and external heterogeneous data. Internal heterogeneous data refers to the data generated by the internal system of the factory park and each device itself, such as: equipment operating status, production indicators, fault alarms, production quality; external heterogeneous data refers to the data received or collected by the factory park from external data sources, such as: supply chain data, market demand data, weather data, and policy and regulatory change data. In addition, multi-source heterogeneous data can also be divided into unstructured data, semi-structured data and structured data according to the data type (i.e., data structure type). Among them, unstructured data refers to data without a specific format and organization, usually in text form, which is difficult to store and manage through traditional databases or tables, such as: equipment logs, sensor data streams, fault alarm texts; structured data refers to data with a clearly defined format and structure, which can be easily stored in a relational database and processed and analyzed through the SQL query language, such as: equipment sensor data tables, production schedules, and production quality reports; semi-structured data is between structured data and unstructured data, with a certain structure but does not conform to the specifications of traditional relational databases, such as: data in XML and JSON formats, and HTML web page text.

[0085] S2, build a data lake based on multi-source heterogeneous data, partition the data lake to obtain a partitioned data lake, and process and analyze the multi-source heterogeneous data based on the partitioned data lake to obtain analytical data.

[0086] In this embodiment, data lake technology is a new technology and new architecture generated in a big data environment. Different from the data storage architecture of traditional databases, data lake supports data storage of all data types, including structured, semi-structured and unstructured, and can retain original data, that is, data lake can store data in its original format until the data is needed.

[0087] According to the preparation data and operation data of industrial equipment, a data lake is constructed, and the data lake is uniformly defined to obtain a defined data lake, wherein the uniform definition includes but is not limited to: defining the target industrial equipment type. The defined data lake is partitioned into the ingestion area, preparation area, analysis area and governance area according to the function to obtain a partitioned data lake, wherein the ingestion area, preparation area, analysis area and governance area each contain two layers, namely the data processing layer and the output storage layer, so that the partitioned data lake can normally access, process and analyze multi-source heterogeneous data, and store the processing and analysis results obtained after the multi-source heterogeneous data is processed and analyzed in each data processing layer of the ingestion area, preparation area, analysis area and governance area. The access rights of each output storage layer of the ingestion area, preparation area, analysis area and governance area can also be defined. For example, the data in the output storage layer of the ingestion area, preparation area and analysis area can be accessed and called not only by the extension module, but also by the governance area.

[0088] The analytical data is a certain data processing and analysis based on multi-source heterogeneous data, which neither changes the data source nor the data structure type of the multi-source heterogeneous data. Therefore, by processing and analyzing internal heterogeneous data and external heterogeneous data based on the partitioned data lake, analytical data of internal heterogeneous data and analytical data of external heterogeneous data can be obtained; by processing and analyzing structured data, semi-structured data, and unstructured data in multi-source heterogeneous data based on the analytical data lake, analytical data of structured data, analytical data of semi-structured data, and analytical data of unstructured data can be obtained.

[0089] With the help of metadata tags, multi-source heterogeneous data of all data types are ingested in their native formats without processing. The ingestion area connects to each data source and extracts multi-source heterogeneous data from each data source to obtain extracted data, and detects whether the multi-source heterogeneous data has changed. Among them, since the native formats of multi-source heterogeneous data of each data type are inconsistent, such as: the native format of structured data can be database tables, CSV files and Excel files, the native format of semi-structured data can be XML, JSON and HTML, and the native format of unstructured data can be txt, img, therefore, the extracted data in multiple formats can be obtained, that is, the extracted data includes structured data in database tables, CSV, Excel file formats, semi-structured data in XML, JSON and HTML formats, and unstructured data in txt, img formats. The ingestion process can be implemented through stream processing, batch processing, real-time processing or hybrid processing technology.

[0090] Data processing is an inevitable step in data analysis. In the preparation area, multi-source heterogeneous data extracted in native format, i.e., extracted data, can be converted into standardized format data according to preset requirements. In other words, multi-source heterogeneous data in different formats can be formatted into one format, such as CSV, JSON, and txt, and converted into a unified format and data type, such as: standardized table data. The data processing process in the preparation area can also be achieved through batch processing and real-time processing technology, that is, batch processing and real-time processing of extracted data can be performed through batch processing and real-time processing technology, so as to enable rapid response and effectively reduce bandwidth usage.

[0091] In the analysis area, standardized format data is effectively read and processed by analysis tools for different data analyses, including but not limited to: reports, statistical analysis, and machine learning algorithms. After data analysis, data reports are generated, wherein data reports include but are not limited to: statistical analysis reports and data intelligence analysis reports.

[0092] Data governance is applied to all areas of the partitioned data lake and is responsible for ensuring data security, data quality, data lifecycle, data access, and metadata management of the partitioned data lake, and preventing the partitioned data lake from deteriorating into a data swamp, where a data swamp refers to a large-scale data repository that is completely inaccessible to users. In other words, the governance area is responsible for data governance of the data collected or generated by the ingestion area, preparation area, analysis area, and governance area of ​​the partitioned data lake. It should be noted that the output storage layers of the ingestion area, preparation area, analysis area, and governance area can periodically clear their respective processing and analysis results according to the clearing strategy, such as: after expanding the analytical data to obtain internal analytical data and external analytical data, and dumping the internal analytical data and external analytical data into the partitioned data lake, the analytical data in the output storage layers of the ingestion area, preparation area, analysis area, and governance area are cleared.

[0093] S3 expands the analytical data to obtain internal analytical data and external analytical data, and dumps the internal analytical data and external analytical data into the partitioned data lake.

[0094] In this embodiment, compared with machine learning and deep learning, knowledge graphs do not require a large amount of training data, providing greater potential for solving problems in many fields. In addition, similarly, the data lake is partitioned into the ingestion area, preparation area, analysis area, and governance area according to the function to obtain a partitioned data lake. The partitioned data lake may also include a storage area, which is used to store dumped internal analysis data and external analysis data.

[0095] It should be noted that the present embodiment can also perform advanced data analysis on the internal analysis data and external analysis data of the storage area through the application layer API call, and the advanced data analysis includes but is not limited to: predictive analysis and intelligent analysis.

[0096] The analytical data of the partitioned data lake based on the knowledge graph expansion is specifically as follows: ① Formal conceptualization indicates the logical design of the knowledge graph. By conducting conceptual analysis on the entities, relationships and attributes in the knowledge graph, the hierarchical structure and semantic relationship between concepts are explored, and relevant concepts and attributes in the analytical data are identified to generate conceptualized analytical data. ② According to the pre-defined domain ontology description, that is, the formal representation of the domain knowledge structure, the concepts, entities and relationships in the domain of interest are captured, and the pre-defined domain ontology structure is combined with the concepts, entities and relationships in the domain of interest to obtain preliminary structured analytical data. Since analytical data includes extracted data, standardized format data, data reports and data security reports, analytical data is essentially heterogeneous data. Through ① and ②, heterogeneous analytical data is converted into consistent graph structure data. ③ Using natural language processing and machine learning technology, semantic information is extracted from the preliminary structured analytical data. The semantic information includes but is not limited to context information and time series information. ④ Generate semantic analysis data with greater depth and breadth by associating and integrating preliminary structured analysis data based on semantic information, including but not limited to entity relationships and attribute value associations, i.e., fused preliminary structured analysis data; and determine the data source type of the fused preliminary structured analysis data based on the data source identification of the fused preliminary structured data itself and the extracted semantic information, and divide the fused preliminary structured analysis data into internal analysis data and external analysis data based on the data source type, wherein the data source type includes internal data source and external data source.

[0097] S4, builds several blocks in the partitioned data lake to generate an initial blockchain, collects the hash values ​​of the changed internal analysis data and external analysis data, and calculates the hash values ​​of each block of the target blockchain based on the hash values ​​of the changed internal analysis data and external analysis data.

[0098] S5, merges the initial blockchain and the changed internal analysis data and external analysis data, uses the hash value of each block of the target blockchain as the header data of the merged initial blockchain, encapsulates the merged initial blockchain, obtains the target blockchain, and compresses the target blockchain.

[0099] In this embodiment, the hash values ​​of the changed internal analysis data and external analysis data are Figure 2 The hash values ​​of the internal analysis data and external analysis data in the target blockchain are Figure 2The hash value of the internal analysis data and the hash value of the external analysis data in the blockchain. Blockchain refers to a chain composed of several blocks, each of which stores certain information. Each block is connected into a chain in the order of their generation, namely the blockchain. The blockchain is stored in all servers. As long as there is a server in the entire system that can work, the entire blockchain is safe. The server is called a blockchain management node in the blockchain system, which provides storage space and computing power support for the entire blockchain system. If you want to modify the information in the blockchain, you must obtain the consent of more than half of the blockchain management nodes and modify the information in all blockchain management nodes. The blockchain management nodes are usually in the hands of different entities, so it is extremely difficult to tamper with the information in the blockchain. The blockchain management node in this embodiment is deployed in the partitioned data lake.

[0100] Compared with traditional storage networks, blockchain has two core features: one is that data is difficult to tamper with, and the other is decentralization. Based on these two core features, the information recorded in blockchain is more authentic and reliable, which can help solve the problem of people's distrust.

[0101] If the ingestion area of ​​the partitioned data lake is the first time that changes in multi-source heterogeneous data are detected after the initial access to the multi-source heterogeneous data of each industrial equipment, then the historical internal analysis data and external analysis data are the initial internal analysis data and external analysis data obtained after the initial access multi-source heterogeneous data is processed and expanded by the partitioned data lake. The historical blockchain is the initial blockchain constructed by generating and connecting several blocks based on the initial internal analysis data and external analysis data.

[0102] It should be noted that if it is not the first time that the ingestion zone detects changes in multi-source heterogeneous data, then the process of generating a historical blockchain based on historical internal analysis data and external analysis data is equivalent to steps S1041-S1043. That is, the last time a change in multi-source heterogeneous data was detected, the target blockchain generated by executing steps S1041-S1043 was the historical blockchain.

[0103] Before generating and compressing the historical blockchain, it also includes deploying a chain compression module and a block production module in the partitioned data lake. The chain compression module and the block production module are the main functional modules of the blockchain management node. Among them, the block production module is used to generate several blocks based on internal analysis data and external analysis data, and merge several blocks to build a blockchain. The chain compression module is used to compress the data in the blockchain.

[0104] It should be noted that this embodiment can selectively compress the historical blockchain, that is, after receiving the compression request, the chain compression module responds to the compression request to start the compression algorithm; after the compression is completed, the chain compression module will also output the compression result.

[0105] Collect the hash values ​​of the changed internal analysis data and external analysis data; timestamp the hash values ​​of the changed internal analysis data and external analysis data and sort them; then fill the changed internal analysis data and external analysis data into the compressed historical blockchain according to the sorting results.

[0106] Due to the characteristics of blockchain, each block contains the hash value of the previous block. By adding the hash value of the previous block to the current block, the blocks can be connected together to form a blockchain, ensuring the immutability of the blockchain. Once an attempt is made to modify the data in the blockchain, the consistency of the hash value will be destroyed, rendering the entire blockchain invalid. Therefore, when a single point failure occurs, it is necessary to access all other blockchain management nodes to recover.

[0107] In this embodiment, the hash values ​​of the changed internal analysis data and external analysis data are used as the hash values ​​of the auxiliary information of the t-th block, and the hash values ​​of each block of the target blockchain are calculated according to Ht=h(ψt) and ψt=(Ht-1,h′(Bt)), where Bt represents the t-th block, h′(Bt) represents the hash value of the auxiliary information of the t-th block, Ht-1 represents the hash value of the t-1-th block, Ht represents the hash value of the t-th block, and h and h′ are two hash functions. The hash values ​​of each block of the target blockchain are used as the header data of each block, and the filled historical blockchain is encapsulated to obtain the target blockchain.

[0108] It needs to be explained that when the hash value of each block of the target blockchain is used as the header data of each block, the filled historical blockchain is encapsulated, and the target blockchain is obtained, the hash value of each block of the target blockchain can be placed in front of the compression depth to facilitate the compression of subsequent blockchains.

[0109] The present embodiment provides a data management method for industrial equipment, which processes and analyzes multi-source heterogeneous data of each industrial equipment based on a partitioned data lake to obtain analysis data, expand the analysis data, obtain internal analysis data and external analysis data, and build a target blockchain in the partitioned data lake based on the internal analysis data and the external analysis data, so as to achieve convenient and efficient management of multi-source heterogeneous data, reduce infrastructure resources and bandwidth consumption, save time costs for data storage and management, help identify multi-faceted data correlations and impacts, improve insight and data analysis capabilities, enhance the effectiveness of decision support, and promote data-driven decision-making and innovative development in various industries.

[0110] Embodiment 3:

[0111] like Figure 3As shown, this embodiment provides a management device for multi-source heterogeneous data, including a collection module 11, a first construction module 12, an expansion module 13, a second construction module 14 and a fusion module 15.

[0112] The acquisition module 11 is used to acquire multi-source heterogeneous data of various industrial equipment.

[0113] The first construction module 12 is connected to the acquisition module 11, and is used to construct a data lake based on multi-source heterogeneous data, partition the data lake to obtain a partitioned data lake, and process and analyze the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data.

[0114] The expansion module 13 is connected to the first construction module 12, and is used to expand the analysis data to obtain internal analysis data and external analysis data, and dump the internal analysis data and the external analysis data into the partition data lake.

[0115] The second construction module 14 is connected to the extension module 13, and is used to construct several blocks in the partitioned data lake to generate an initial blockchain, collect the hash values ​​of the changed internal analysis data and the external analysis data, and calculate the hash values ​​of each block of the target blockchain according to the hash values ​​of the changed internal analysis data and the external analysis data.

[0116] The fusion module 15 is connected to the second building module connection 14, and is used to fuse the initial blockchain and the changed internal analysis data and external analysis data, use the hash value of each block of the target blockchain as the header data of the fused initial blockchain, encapsulate the fused initial blockchain, obtain the target blockchain, and compress the target blockchain.

[0117] This embodiment provides a data management device for industrial equipment, which processes and analyzes multi-source heterogeneous data of each industrial equipment based on a partitioned data lake to obtain analysis data, expand the analysis data, obtain internal analysis data and external analysis data, and build a target blockchain in the partitioned data lake based on the internal analysis data and the external analysis data, so as to achieve convenient and efficient management of multi-source heterogeneous data, reduce infrastructure resources and bandwidth consumption, save time costs for data storage and management, help identify multi-faceted data correlations and impacts, improve insight and data analysis capabilities, enhance the effectiveness of decision support, and promote data-driven decision-making and innovative development in various industries.

[0118] Embodiment 4:

[0119] like Figure 4As shown, this embodiment also provides a computer device, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the multi-source heterogeneous data management method of embodiment 1 or embodiment 2 is implemented.

[0120] The memory 10 includes at least one type of readable storage medium, wherein the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory, magnetic memory, disk, optical disk, and card-type memory includes but is not limited to: SD (SecureDigital, a proprietary non-volatile memory card format) or DX memory. The memory 10 can be an internal storage unit of a computer device, such as: a hard disk of a computer device. The memory 10 can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (FlashCard). Further, the memory 10 can also include both an internal storage unit of a computer device and an external storage device. The memory 10 can be used not only to store application software and various types of data installed in the computer device, but also to temporarily store data that has been output or is to be output.

[0121] The processor 20 can be an electronic control unit (ECU, also known as a vehicle computer), a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the memory 10 or process data, such as executing an access restriction program.

[0122] Embodiment 5:

[0123] like Figure 5 As shown, this embodiment also provides a management device for multi-source heterogeneous data, including an acquisition module 51, an analysis module 52, an expansion module 53 and a construction module 54. The acquisition module 51 is used to acquire multi-source heterogeneous data of each device. The analysis module 52 is connected to the acquisition module 51 and is used to process and analyze the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data. The expansion module 53 is connected to the analysis module 52 and is used to expand the analysis data to obtain internal analysis data and external analysis data, and dump the internal analysis data and external analysis data into the partitioned data lake. The construction module 54 is connected to the expansion module 53 and is used to build a target blockchain in the partitioned data lake.

[0124] Specifically, the acquisition module 51 also includes: a collection unit 511 and a marking unit 512. The collection unit 511 is used to collect preparation data and operation data of each device from various data sources. The marking unit 512 is used to perform metadata marking on the preparation data and the operation data respectively to obtain multi-source heterogeneous data of each device.

[0125] Optionally, the multi-source heterogeneous data management device also includes: a partitioning module 55, which is used to build a data lake based on the multi-source heterogeneous data, and uniformly define the data lake to obtain a defined data lake. The construction module 55 is also used to partition the defined data lake into an ingestion area, a preparation area, an analysis area, and a governance area to obtain a partitioned data lake.

[0126] Specifically, the analysis module 52 includes: an extraction unit 521, a preparation unit 522, an analysis unit 523 and a governance unit 524. The extraction unit 521 is used to extract the multi-source heterogeneous data based on the ingestion area to obtain the extracted data and detect changes in the multi-source heterogeneous data. The preparation unit 522 is used to convert the extracted data into a standardized format based on the preparation area to obtain standardized format data. The analysis unit 523 is used to analyze the standard formatted data based on the analysis area and generate a data report. The governance unit 524 is used to perform data governance on the extracted data, the standardized format data and the data report based on the governance area and generate a security data report.

[0127] Specifically, the extension module 53 includes: a conceptualization unit 531, a structuring unit 532, an extraction unit 533 and a fusion unit 534. The conceptualization unit 531 is used to formally conceptualize the analysis data based on the knowledge graph to obtain conceptualized analysis data. The structuring unit 532 is used to structure the conceptualized analysis data according to a predefined domain to obtain preliminary structured analysis data. The extraction unit 533 is used to extract semantic information of the preliminary structured analysis data. The fusion unit 534 is used to fuse and divide the preliminary structured analysis data according to the semantic information to obtain internal analysis data and external analysis data.

[0128] Specifically, the construction module 54 includes: a generation unit 541, a compression unit 542 and a filling unit 543. The generation unit 541 is used to generate a historical blockchain based on historical internal analysis data and external analysis data. The compression unit 542 is used to compress the historical blockchain. The filling unit 543 is used to fill and encapsulate the compressed historical blockchain according to the changed internal analysis data and external analysis data to obtain the target blockchain.

[0129] Specifically, the filling unit 543 includes: a collection subunit, a sorting subunit, a filling subunit and an encapsulation subunit. The collection subunit is used to collect the hash values ​​of the changed internal analysis data and the external analysis data. The sorting subunit is used to timestamp the hash values ​​and sort them. The filling subunit is used to fill the changed internal analysis data and the external analysis data into the compressed historical blockchain according to the sorting results. The encapsulation subunit is used to encapsulate the filled historical blockchain to obtain the target blockchain.

[0130] It can be understood that the multi-source heterogeneous data management device provided above executes the multi-source heterogeneous data management method corresponding to the embodiment 1 provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the scheme corresponding to the multi-source heterogeneous data management method of the embodiment 1 above, and will not be repeated here.

[0131] Embodiment 6:

[0132] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the multi-source heterogeneous data management method in the above-mentioned embodiment 1.

[0133] Embodiment 7:

[0134] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for managing multi-source heterogeneous data in the above-mentioned embodiment 1 is implemented.

[0135] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for managing multi-source heterogeneous data, characterized in that: include: Acquire multi-source heterogeneous data from each device; Processing and analyzing the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data; Expanding the analysis data to obtain internal analysis data and external analysis data, and dumping the internal analysis data and the external analysis data into the partitioned data lake; A target blockchain is built in the partitioned data lake.

2. The method for managing multi-source heterogeneous data according to claim 1, characterized in that: The obtaining of multi-source heterogeneous data of each device specifically includes: Collect preparation data and operation data of each device from various data sources; Metadata marking is performed on the preparation data and the operation data respectively to obtain multi-source heterogeneous data of each device.

3. The method for managing multi-source heterogeneous data according to claim 1, characterized in that: After acquiring the multi-source heterogeneous data of each device and before processing and analyzing the multi-source heterogeneous data based on the partitioned data lake to obtain the analysis data, the method further includes: Building a data lake based on the multi-source heterogeneous data, and uniformly defining the data lake to obtain a defined data lake; The defined data lake is partitioned into an ingestion zone, a preparation zone, an analysis zone, and a governance zone to obtain a partitioned data lake.

4. The method for managing multi-source heterogeneous data according to claim 1, characterized in that: The analytical data includes extraction data, standardized format data, data reports and safety data reports, The processing and analysis of the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data specifically includes: Extracting the multi-source heterogeneous data based on the ingestion area to obtain extracted data, and detecting changes in the multi-source heterogeneous data; Converting the extracted data into a standardized format based on the preparation area to obtain standardized format data; Analyzing the standard formatted data based on the analysis area and generating a data report; Data governance is performed on the extracted data, the standardized format data, and the data report based on the governance zone, and a secure data report is generated.

5. The method for managing multi-source heterogeneous data according to claim 1, characterized in that: The expansion of the analysis data to obtain internal analysis data and external analysis data specifically includes: Based on the knowledge graph, the analysis data is formally conceptualized to obtain conceptualized analysis data; Structuring the conceptual analysis data according to pre-defined domains to obtain preliminary structured analysis data; Extracting semantic information from the preliminary structured analysis data; According to the semantic information, the preliminary structured analysis data is integrated and divided to obtain internal analysis data and external analysis data.

6. The method for managing multi-source heterogeneous data according to claim 1, characterized in that: The internal analysis data and external analysis data include historical internal analysis data and external analysis data, as well as changed internal analysis data and external analysis data. The constructing of the target blockchain in the partitioned data lake specifically includes: Generate historical blockchain based on historical internal and external analysis data; Compress the historical blockchain; According to the changed internal analysis data and external analysis data, the compressed historical blockchain is filled and encapsulated to obtain the target blockchain.

7. The method for managing multi-source heterogeneous data according to claim 6, characterized in that: The method of filling and encapsulating the compressed historical blockchain according to the changed internal analysis data and external analysis data to obtain the target blockchain specifically includes: Collect the hash values ​​of the changed internal analysis data and external analysis data; Timestamping the hash values ​​and sorting them; According to the sorting results, the changed internal analysis data and external analysis data are filled into the compressed historical blockchain; Encapsulate the filled historical blockchain to obtain the target blockchain.

8. A management device for multi-source heterogeneous data, characterized in that: include: Get modules, analyze modules, extend modules, and build modules, The acquisition module is used to obtain multi-source heterogeneous data from each device. The analysis module is connected to the acquisition module and is used to process and analyze the multi-source heterogeneous data based on the partitioned data lake to obtain analysis data. An expansion module, connected to the analysis module, is used to expand the analysis data to obtain internal analysis data and external analysis data, and dump the internal analysis data and external analysis data into the partition data lake. A building module, connected with the extension module, is used to build a target blockchain in the partitioned data lake.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement a method for managing multi-source heterogeneous data as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a method for managing multi-source heterogeneous data as described in any one of claims 1 to 7 is implemented.