A big data standardization method and system based on data integration

Abstract

The present invention provides a big data standardization method and system based on data integration, including: using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm to convert original data from different data sources into intermediate format data; using quantum state encoding to generate a quantum superposition state of each field in the intermediate format data and a node in a knowledge graph; using a quantum annealing algorithm based on the quantum superposition state to solve the optimal mapping node of the field in the knowledge graph; based on the optimal mapping node combined with the semantic constraints and standard format of the node, the field is standardized and mapped to obtain standardized data. In the present invention, multi-source data is integrated and parsed into intermediate format data through a dynamic format adapter, thereby solving the problem of data heterogeneity, retaining multiple mapping possibilities simultaneously based on the quantum superposition state, and obtaining the global optimal solution through rapid parallel search through the quantum annealing algorithm, thereby improving the accuracy and efficiency of node standardized mapping.

Description

Technical Field

[0001] The present invention relates to the technical field of data standardization, and in particular to a big data standardization method and system based on data integration. Background Art

[0002] With the rapid development of information technology, big data has become a vital resource in modern society. The core value of big data lies in supporting decision-making through the analysis and mining of massive amounts of data. However, the diversity and complexity of big data also present numerous challenges, particularly in data integration and interoperability. Data integration refers to the integration of data from diverse sources into a unified data view, while data interoperability refers to the ability to seamlessly exchange and share data between different systems. In big data environments, data sources are often heterogeneous, including structured, semi-structured, and unstructured data. This heterogeneity makes data integration and interoperability particularly complex.

[0003] Traditional data integration methods primarily include ETL (Extract, Transform, Load) tools, which extract data from diverse data sources, transform and cleanse it, and then load it into the target database. However, with the rapid increase in the diversity and volume of data sources, traditional ETL tools face numerous challenges when dealing with heterogeneous data sources.

[0004] The data formats of different data sources vary greatly, which makes format conflicts easy to occur during the data integration process; and due to the independence between data sources, the same data may exist in different forms in different data sources, resulting in data redundancy and inconsistency, which not only increases the cost of data storage and processing, but also may affect the accuracy of data analysis; some data may be lost during the data collection process, which will affect the integrity of the data and the reliability of the analysis results. Summary of the Invention

[0005] To overcome the above-mentioned deficiencies of the prior art, the present invention proposes a big data standardization method based on data integration, comprising:

[0006] Obtain raw data from multiple different data sources;

[0007] Using a dynamic format adapter to convert the original data into intermediate format data based on data conversion mapping rules combined with a reinforcement learning Schema algorithm;

[0008] Based on each field in the intermediate format data, quantum state encoding is used to generate a quantum superposition state of the field and a node in the knowledge graph; based on the quantum superposition state, a quantum annealing algorithm is used to solve the optimal mapping node of the field in the knowledge graph;

[0009] Based on the optimal mapping node for each field in the intermediate format data, combined with the semantic constraints and standard format of the nodes in the knowledge graph, the data corresponding to each field in the intermediate format data is standardized and mapped to obtain standardized data.

[0010] Optionally, the knowledge graph construction process includes:

[0011] Determine the core entities and attributes of the external multi-source data as nodes of the initial knowledge graph, wherein the nodes include entity nodes and attribute nodes;

[0012] Based on each sample field in the external multi-source data, determining a mapping node for the sample field using the semantic similarity between the sample field and each of the nodes, and mapping the sample field to the initial knowledge graph based on the mapping node;

[0013] In the initial knowledge graph after mapping and filling, the semantic constraints and standard formats of the attribute nodes are set to obtain a knowledge graph for data conversion.

[0014] Optionally, the generating of a quantum superposition state of the field and a node in the knowledge graph by using quantum state encoding includes:

[0015] Based on the semantic similarity between the field and each node in the knowledge graph and in combination with a similarity threshold, selecting multiple candidate mapping nodes for the field from the knowledge graph;

[0016] quantizing the semantic relationship between the field and each candidate mapping node into a probability amplitude using quantum state coding;

[0017] Combining the probability amplitude between the field and each of the candidate mapping nodes, a quantum superposition state of the field and the nodes in the knowledge graph is obtained.

[0018] Optionally, the step of solving the optimal mapping node of the field in the knowledge graph using a quantum annealing algorithm based on the quantum superposition state includes:

[0019] Utilizing the quantum tunneling effect in the quantum degeneration algorithm, based on the quantum superposition state, preset temperature parameters and energy parameters, the probability amplitude of the candidate mapping node in the quantum superposition state is iteratively updated multiple times;

[0020] After each iterative update, the temperature parameter is actively reduced according to a preset attenuation coefficient; the iterative update is stopped until the temperature parameter drops to a threshold, and the candidate node with the highest probability amplitude in the quantum superposition state when the iterative update is stopped is used as the optimal mapping node.

[0021] Optionally, the process of calculating the semantic similarity between the field and each node in the knowledge graph includes:

[0022] Extracting semantic vectors of the fields using named entity recognition technology;

[0023] By calculating the cosine similarity between the semantic vector of the field and the semantic vector of each node in the knowledge graph, the semantic similarity between the field and each of the nodes is obtained.

[0024] Optionally, the converting of the original data into intermediate format data using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm includes:

[0025] Identifying the format of the original data using a dynamic format adapter and parsing the initial field type in the original data;

[0026] Converting the original data into an internal data structure using a parser based on the format, recorded as initial parsed data;

[0027] Correcting the initial field type based on the initial parsed data using a reinforcement learning schema algorithm to obtain the field type in the original data;

[0028] Based on the data conversion mapping rules and the machine learning algorithm, the original data is converted into intermediate format data in combination with the field type.

[0029] Optionally, converting the original data into intermediate format data based on the data conversion mapping rules and the machine learning algorithm in combination with the field type includes:

[0030] Determining, based on the field type of the original data and the predefined mapping rule, original data for which the mapping rule exists and original data for which the mapping rule does not exist;

[0031] Based on the field type of the original data and the mapping rule, converting the original data with the mapping rule into intermediate format data;

[0032] A machine learning algorithm is used to perform semantic inference and conversion on the original data for which no mapping rules exist based on the field type of the original data to obtain intermediate format data.

[0033] Optionally, before converting the original data into intermediate format data using the dynamic format adapter based on the data conversion mapping rules and the machine learning algorithm, the method further includes:

[0034] Edge computing technology is used to extract and filter the raw data from the multiple different data sources to obtain processed raw data.

[0035] Based on the same inventive concept, the present invention proposes a big data standardization system based on data integration, comprising:

[0036] Data acquisition module, used to obtain raw data from multiple different data sources;

[0037] An intermediate format conversion module, configured to convert the original data into intermediate format data using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm;

[0038] A node solving module is configured to generate, based on each field in the intermediate format data, a quantum superposition state of the field and a node in the knowledge graph using quantum state encoding; and based on the quantum superposition state, use a quantum annealing algorithm to solve the optimal mapping node of the field in the knowledge graph;

[0039] A standardized mapping module is used to perform standardized mapping on the data corresponding to each field in the intermediate format data based on the optimal mapping node for each field in the intermediate format data, combined with the semantic constraints and standard format of the nodes in the knowledge graph, to obtain standardized data.

[0040] Optionally, the node solving module is specifically used to:

[0041] Determine the core entities and attributes of the external multi-source data as nodes of the initial knowledge graph, wherein the nodes include entity nodes and attribute nodes;

[0042] Based on each sample field in the external multi-source data, determining a mapping node for the sample field using the semantic similarity between the sample field and each of the nodes, and mapping the sample field to the initial knowledge graph based on the mapping node;

[0043] In the initial knowledge graph after mapping and filling, the semantic constraints and standard formats of the attribute nodes are set to obtain a knowledge graph for data conversion.

[0044] Optionally, the node solving module is specifically used to:

[0045] Based on the semantic similarity between the field and each node in the knowledge graph and in combination with a similarity threshold, selecting multiple candidate mapping nodes for the field from the knowledge graph;

[0046] quantizing the semantic relationship between the field and each candidate mapping node into a probability amplitude using quantum state coding;

[0047] Combining the probability amplitude between the field and each of the candidate mapping nodes, a quantum superposition state of the field and the nodes in the knowledge graph is obtained.

[0048] Optionally, the node solving module is specifically used to:

[0049] Utilizing the quantum tunneling effect in the quantum degeneration algorithm, based on the quantum superposition state, preset temperature parameters and energy parameters, the probability amplitude of the candidate mapping node in the quantum superposition state is iteratively updated multiple times;

[0050] After each iterative update, the temperature parameter is actively reduced according to a preset attenuation coefficient; the iterative update is stopped until the temperature parameter drops to a threshold, and the candidate node with the highest probability amplitude in the quantum superposition state when the iterative update is stopped is used as the optimal mapping node.

[0051] Optionally, the node solving module is specifically used to:

[0052] Extracting semantic vectors of the fields using named entity recognition technology;

[0053] By calculating the cosine similarity between the semantic vector of the field and the semantic vector of each node in the knowledge graph, the semantic similarity between the field and each of the nodes is obtained.

[0054] Optionally, the intermediate format conversion module is specifically configured to:

[0055] Identifying the format of the original data using a dynamic format adapter and parsing the initial field type in the original data;

[0056] Converting the original data into an internal data structure using a parser based on the format, recorded as initial parsed data;

[0057] Correcting the initial field type based on the initial parsed data using a reinforcement learning schema algorithm to obtain the field type in the original data;

[0058] Based on the data conversion mapping rules and the machine learning algorithm, the original data is converted into intermediate format data in combination with the field type.

[0059] Optionally, the intermediate format conversion module is specifically configured to:

[0060] Determining, based on the field type of the original data and the predefined mapping rule, original data for which the mapping rule exists and original data for which the mapping rule does not exist;

[0061] Based on the field type of the original data and the mapping rule, converting the original data with the mapping rule into intermediate format data;

[0062] A machine learning algorithm is used to perform semantic inference and conversion on the original data for which no mapping rules exist based on the field type of the original data to obtain intermediate format data.

[0063] Optionally, the system further includes an edge computing module, configured to:

[0064] Edge computing technology is used to extract and filter the raw data from the multiple different data sources to obtain processed raw data.

[0065] In another aspect, the present application further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0066] The memory is used to store one or more programs;

[0067] When the one or more programs are executed by the at least one processor, a big data standardization method based on data integration as described above is implemented.

[0068] On the other hand, the present application also provides a computer-readable storage medium having an execution program stored thereon, which, when executed, implements a big data standardization method based on data integration as described above.

[0069] Compared with the closest prior art, the present invention has the following beneficial effects:

[0070] The present invention provides a big data standardization method and system based on data integration, comprising: obtaining raw data from multiple different data sources; using a dynamic format adapter to convert the raw data into intermediate format data based on data conversion mapping rules combined with a reinforcement learning Schema algorithm; based on each field in the intermediate format data, using quantum state encoding to generate a quantum superposition state of the field and a node in a knowledge graph; based on the quantum superposition state, using a quantum annealing algorithm to solve the optimal mapping node of the field in the knowledge graph; based on the optimal mapping node of each field in the intermediate format data, combining the semantic constraints and standard format of the nodes in the knowledge graph, The data corresponding to each field in the data is standardized and mapped to obtain standardized data; the dynamic format adapter in the present invention integrates multi-source data and parses it into intermediate format data that can be processed by the system, solving the problem of data heterogeneity; the schema algorithm solves the structural deviation between the original data and the converted format by correcting the field type, thereby improving the accuracy of the intermediate format data; the quantum superposition state can retain multiple mapping possibilities at the same time, and the quantum annealing algorithm can jump out of the local optimum and quickly obtain the global optimal solution through efficient parallel search capabilities, thereby improving the accuracy and efficiency of node mapping; based on the best mapping node, semantic constraints and standard format mapping are performed in the knowledge graph to achieve data standardization processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1A flowchart of a big data standardization method based on data integration provided by the present invention;

[0072] Figure 2 A schematic diagram of the structure of a big data standardization system based on data integration provided by the present invention;

[0073] Figure 3 This is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0074] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] Example 1

[0076] The present invention provides a big data standardization method based on data integration, such as Figure 1 Shown, including:

[0077] S1. Obtain raw data from multiple different data sources;

[0078] S2. Using a dynamic format adapter to convert the original data into intermediate format data based on data conversion mapping rules combined with a reinforcement learning Schema algorithm;

[0079] S3. Based on each field in the intermediate format data, generate a quantum superposition state of the field and a node in the knowledge graph using quantum state encoding; based on the quantum superposition state, use a quantum annealing algorithm to solve the optimal mapping node of the field in the knowledge graph;

[0080] S4. Based on the optimal mapping node for each field in the intermediate format data, combined with the semantic constraints and standard format of the nodes in the knowledge graph, the data corresponding to each field in the intermediate format data is standardized and mapped to obtain standardized data.

[0081] In step S1, the raw data from different data sources is obtained. It is responsible for extracting data from different data sources, supporting multiple data formats and protocols, and ensuring the wide compatibility of data. Specifically:

[0082] Utilizing a plurality of data source access devices distributedly arranged near the plurality of different data sources, obtaining the raw data of the plurality of different data sources; each data source access module corresponds to one or more of the data sources;

[0083] The data source access devices are distributed to the edge nodes, and the original data from the multiple different data sources are extracted and filtered using edge computing technology (multiple edge computing devices are set up) to obtain the processed original data, realize real-time data access and preliminary processing, and reduce the system load.

[0084] In the IoT environment, sensors may frequently send similar data points. By distributing multiple data source access devices near the data source, the edge node first performs preliminary processing on the data (such as filtering, aggregation, compression, etc.), and only sends key information to the dynamic format adapter. This can significantly reduce the amount of data that needs to be transmitted to the dynamic format adapter, thereby reducing network bandwidth requirements and transmission costs, and accelerating transmission efficiency.

[0085] In step S2, the original data is converted into intermediate format data using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm.

[0086] The dynamic format adapter supports real-time learning and adaptation of new data formats. Through machine learning algorithms, the system can automatically learn the format rules of new data sources and generate corresponding conversion logic. The design process of the dynamic format adapter is as follows:

[0087] Based on the data format conversion requirements of different data sources, a modular architecture is designed, including a data format recognition module, a conversion logic generation module, and a data conversion module.

[0088] A machine learning algorithm is introduced into the conversion logic generation module to predict the optimal conversion strategy by learning from historical data. The machine learning algorithm is continuously updated as more data formats are added, realizing the function of a dynamic format adapter supporting real-time learning and adaptation of new data formats.

[0089] S21. Using a dynamic format adapter to identify the format of the original data and parse the initial field type in the original data; based on the format, using a parser to convert the original data into an internal data structure, which is recorded as initial parsed data.

[0090] Among them, identifying the format of the original data is to call the parser corresponding to different data formats, parse the original data, and convert the original data into an internal data structure (such as a dictionary, data frame) that can be operated by the program, providing a basis for subsequent Schema inference and field extraction; by parsing the initial field type in the original data, that is, predicting the field type through format features (such as the value wrapped in quotation marks is a string), the initial search space of the reinforcement learning schema algorithm (reinforcement learning architecture algorithm) is reduced, and the data processing efficiency is improved.

[0091] S22. Modify the initial field type based on the initial parsed data using a reinforcement learning schema algorithm to obtain the field type in the original data.

[0092] Through the reinforcement learning schema algorithm, the field types and nested relationships in the original data are corrected, and the structural deviation between the original data and the format to be converted is resolved. This provides a data structure reference for subsequent rule engines and machine learning, ensures that the conversion logic corresponds to the data structure, and improves the accuracy of the intermediate format data.

[0093] S23. Based on the data conversion mapping rules and the machine learning algorithm, the raw data is converted into intermediate format data in combination with the field type, including:

[0094] Determining, based on the field type of the original data and the predefined mapping rule, original data for which the mapping rule exists and original data for which the mapping rule does not exist;

[0095] Based on the field type of the original data and the mapping rule, converting the original data with the mapping rule into intermediate format data;

[0096] A machine learning algorithm is used to perform semantic inference and conversion on the original data for which no mapping rules exist based on the field type of the original data to obtain intermediate format data.

[0097] Among them, the data structures of different data sources may vary greatly, and the intermediate format needs to be compatible with the structures of different data sources; if the data volume is large, the intermediate format needs to be designed as an efficient structure to reduce the overhead of conversion and transmission.

[0098] For raw data with clear mapping rules, it can be converted directly with high conversion efficiency. For raw data without mapping rules, machine learning algorithms can be used to intelligently parse and convert data, solve problems with complex or unstructured formats, and improve the coverage and accuracy of data conversion. Combining rules and machine learning can process structured and unstructured data at the same time, adapting to the diversity of multiple data sources.

[0099] Dynamic format adapters are used to extract raw data from various data sources and convert it into an intermediate format that the system can handle for subsequent processing. This approach addresses heterogeneity in data sources, such as differing protocols, encodings, and structures, ensuring that data can be received by the system. This improves data access compatibility, supporting multiple source data types and providing high-quality input for subsequent modules. The resulting intermediate format data can be correctly parsed and processed by subsequent modules, preserving the accuracy of the original data.

[0100] In step S3, the pre-construction process of the knowledge graph is as follows:

[0101] Obtain semantic information of multiple sample fields in external multi-source data;

[0102] Defining core entities and attributes based on the external multi-source data, and determining the core entities and attributes as nodes of an initial knowledge graph, wherein the nodes include entity nodes and attribute nodes;

[0103] Based on each of the sample fields, obtaining the semantic similarity between the sample field and each of the nodes; determining a mapping node using the semantic similarity between the sample field and each of the nodes, and mapping the sample field to the initial knowledge graph based on the mapping node;

[0104] and representing the semantic relationship between the field name and the core entity and attribute as an edge, and filling the initial knowledge graph based on the nodes and edges;

[0105] In the initial knowledge graph after mapping and filling, the semantic constraints and standard formats of the attribute nodes are set to obtain a knowledge graph for data conversion.

[0106] Semantic constraints in the knowledge graph are used to ensure that field values ​​comply with business logic rules. They do not directly change the format but affect the legality of the conversion.

[0107] For example: verifying data types (e.g., "price" must be numeric, and an error will be reported if the original value is "100 yuan"); or verifying business rules (e.g., "date of birth ≤ current date", and an exception will be marked if the date is in the future).

[0108] Only when the field value passes the semantic constraints will it enter the format conversion stage.

[0109] S31. Before performing quantum state encoding, extract multiple fields from the intermediate format data, wherein the fields include field names and field values. Common types of intermediate format data include:

[0110] JSON (JavaScript Object Notation, a lightweight data exchange format) data: key-value pair structure, field name is the key. The field name is obtained by parsing the first-level key of the JSON data, and the field value is directly read from the value corresponding to the key.

[0111] XML (eXtensible Markup Language) data: tag structure, field name is the tag name, obtained by parsing the sub-tags under the root node of the XML data, extracting the text content within the tag, and obtaining the field value.

[0112] CSV (Comma-Separated Values, a text file format) data: A table structure with field names usually in the first row. Field names are obtained by reading the first row of the file, and field values ​​are obtained from the corresponding column positions in subsequent rows.

[0113] S32. Generate a quantum superposition state of each field in the intermediate format data and a node in the knowledge graph using quantum state encoding.

[0114] For each of the fields in the intermediate format data:

[0115] a. Calculate the semantic similarity between the field and each node in the knowledge graph, and select multiple candidate mapping nodes for the field from the knowledge graph based on the semantic similarity between the field and each node in the knowledge graph and a similarity threshold. This includes:

[0116] Extracting semantic vectors of the fields using named entity recognition technology or word vector models;

[0117] By calculating the cosine similarity between the semantic vector of the field and the semantic vector of each node in the knowledge graph, the semantic similarity between the field and each of the nodes is obtained.

[0118] Nodes in the knowledge graph whose semantic similarity is greater than a preset similarity threshold are selected as candidate mapping nodes.

[0119] By screening the nodes in the knowledge graph based on semantic similarity, the computational complexity of the subsequent quantum matching process is significantly reduced, and data processing efficiency is improved.

[0120] b. Quantify the semantic relationship between the field and each candidate mapping node into a probability amplitude using quantum state encoding; and obtain the quantum superposition state of the field and the nodes in the knowledge graph by combining the probability amplitudes between the field and each candidate mapping node.

[0121] By using quantum superposition states to simultaneously represent the association probability between a field and multiple candidate nodes, the inefficiency of traditional methods requiring multiple serial matching is avoided, significantly improving the processing speed of complex semantic scenarios (such as polysemy of a word).

[0122] S33. Based on the quantum superposition state, use a quantum annealing algorithm to find the best mapping node for the field in the knowledge graph. Specifically:

[0123] For each of the fields in the intermediate format data:

[0124] The probability amplitude in the quantum superposition state corresponding to the field is used as the initial state;

[0125] Utilizing the quantum tunneling effect in the quantum degeneration algorithm, based on the quantum superposition state, preset temperature parameters (initial high temperature parameters) and energy parameters (based on the semantic similarity calculation rules of the semantic vector), the probability amplitude of the candidate mapping nodes in the quantum superposition state is iteratively updated multiple times;

[0126] After each iterative update, the temperature parameter is actively reduced according to a preset attenuation coefficient; the iterative update is stopped until the temperature parameter drops to a threshold, and the candidate node with the highest probability amplitude in the quantum superposition state when the iterative update is stopped is used as the optimal mapping node.

[0127] Each iteration adjusts the probability amplitude of the candidate mapping nodes according to the semantic similarity. After multiple iterative updates, the system converges to a stable state and retains the node with the highest probability amplitude as the best mapping node.

[0128] Quantum annealing uses the quantum tunneling effect to simultaneously explore the matching possibilities of multiple candidate mapping nodes, preventing traditional algorithms from falling into local optimal solutions. By combining the adjustment of probability amplitude and the energy function (semantic similarity), it can quickly converge to the most logical mapping node while retaining business semantic relevance (such as the ambiguity of the same field in different scenarios).

[0129] In step S4, the formats of multiple fields in the intermediate format data are automatically corrected according to the semantic constraints and standard formats corresponding to the optimal mapping nodes to obtain standardized data, so that it can be used for data integration and interoperability of multi-source data.

[0130] Semantic constraints are used to ensure that field values ​​comply with business logic rules. Field values ​​that have passed the semantic constraints are rewritten in a standard format. That is, illegal values ​​can be filtered through semantic constraints. If the field value fails to pass the semantic constraints, the format conversion is terminated and an error is recorded.

[0131] Example 2

[0132] Based on the same inventive concept, the present invention also provides a big data standardization system based on data integration, such as Figure 2 Shown, including:

[0133] Data acquisition module, used to obtain raw data from multiple different data sources;

[0134] An intermediate format conversion module, configured to convert the original data into intermediate format data using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm;

[0135] A node solving module is configured to generate, based on each field in the intermediate format data, a quantum superposition state of the field and a node in the knowledge graph using quantum state encoding; and based on the quantum superposition state, use a quantum annealing algorithm to solve the optimal mapping node of the field in the knowledge graph;

[0136] A standardized mapping module is used to perform standardized mapping on the data corresponding to each field in the intermediate format data based on the optimal mapping node for each field in the intermediate format data, combined with the semantic constraints and standard format of the nodes in the knowledge graph, to obtain standardized data.

[0137] In a possible implementation, the node solving module is specifically used to:

[0138] Determine the core entities and attributes of the external multi-source data as nodes of the initial knowledge graph, wherein the nodes include entity nodes and attribute nodes;

[0139] Based on each sample field in the external multi-source data, determining a mapping node for the sample field using the semantic similarity between the sample field and each of the nodes, and mapping the sample field to the initial knowledge graph based on the mapping node;

[0140] In the initial knowledge graph after mapping and filling, the semantic constraints and standard formats of the attribute nodes are set to obtain a knowledge graph for data conversion.

[0141] In a possible implementation, the node solving module is specifically used to:

[0142] Based on the semantic similarity between the field and each node in the knowledge graph and in combination with a similarity threshold, selecting multiple candidate mapping nodes for the field from the knowledge graph;

[0143] quantizing the semantic relationship between the field and each candidate mapping node into a probability amplitude using quantum state coding;

[0144] Combining the probability amplitude between the field and each of the candidate mapping nodes, a quantum superposition state of the field and the nodes in the knowledge graph is obtained.

[0145] In a possible implementation, the node solving module is specifically used to:

[0146] Utilizing the quantum tunneling effect in the quantum degeneration algorithm, based on the quantum superposition state, preset temperature parameters and energy parameters, the probability amplitude of the candidate mapping node in the quantum superposition state is iteratively updated multiple times;

[0147] After each iterative update, the temperature parameter is actively reduced according to a preset attenuation coefficient; the iterative update is stopped until the temperature parameter drops to a threshold, and the candidate node with the highest probability amplitude in the quantum superposition state when the iterative update is stopped is used as the optimal mapping node.

[0148] In a possible implementation, the node solving module is specifically used to:

[0149] Extracting semantic vectors of the fields using named entity recognition technology;

[0150] By calculating the cosine similarity between the semantic vector of the field and the semantic vector of each node in the knowledge graph, the semantic similarity between the field and each of the nodes is obtained.

[0151] In a possible implementation, the intermediate format conversion module is specifically configured to:

[0152] Identifying the format of the original data using a dynamic format adapter and parsing the initial field type in the original data;

[0153] Converting the original data into an internal data structure using a parser based on the format, recorded as initial parsed data;

[0154] Correcting the initial field type based on the initial parsed data using a reinforcement learning schema algorithm to obtain the field type in the original data;

[0155] Based on the data conversion mapping rules and the machine learning algorithm, the original data is converted into intermediate format data in combination with the field type.

[0156] In a possible implementation, the intermediate format conversion module is specifically configured to:

[0157] Determining, based on the field type of the original data and the predefined mapping rule, original data for which the mapping rule exists and original data for which the mapping rule does not exist;

[0158] Based on the field type of the original data and the mapping rule, converting the original data with the mapping rule into intermediate format data;

[0159] A machine learning algorithm is used to perform semantic inference and conversion on the original data for which no mapping rules exist based on the field type of the original data to obtain intermediate format data.

[0160] In one possible implementation, the system further includes an edge computing module configured to:

[0161] Edge computing technology is used to extract and filter the raw data from the multiple different data sources to obtain processed raw data.

[0162] Example 3

[0163] like Figure 3As shown, the present invention also provides an electronic device, which may be a computer, a single-chip microcomputer, a smart mobile device, or the like. The electronic device in this embodiment may include a processor, a memory, a transceiver component, and the like. The memory, processor, and transceiver component are connected via a bus; the memory may be used to store an execution program, which may include instructions; and the processor may be used to execute the instructions stored in the memory. The memory may also be used to store data, which may be accessed and / or modified during the execution of the instructions.

[0164] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a big data standardization method based on data integration in the above embodiment.

[0165] Example 4

[0166] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a big data standardization method based on data integration in the above embodiment.

[0167] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art may still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims.

Claims

1. A big data standardization method based on data integration, characterized in that: include: Obtain raw data from multiple different data sources; Using a dynamic format adapter to convert the original data into intermediate format data based on data conversion mapping rules combined with a reinforcement learning Schema algorithm; Based on each field in the intermediate format data, quantum state coding is used to generate a quantum superposition state of the field and a node in the knowledge graph, including: based on the semantic similarity between the field and each node in the knowledge graph, combined with a similarity threshold, selecting multiple candidate mapping nodes for the field from the knowledge graph; using quantum state coding to quantify the semantic relationship between the field and each candidate mapping node into a probability amplitude; combining the probability amplitude between the field and each candidate mapping node to obtain a quantum superposition state of the field and the node in the knowledge graph; Based on the quantum superposition state, a quantum annealing algorithm is used to solve the optimal mapping node of the field in the knowledge graph, including: utilizing the quantum tunneling effect in the quantum annealing algorithm, based on the quantum superposition state, preset temperature parameters and energy parameters, iteratively updating the probability amplitude of the candidate mapping node in the quantum superposition state multiple times; after each iterative update, the temperature parameter is actively reduced according to a preset attenuation coefficient; the iterative update is stopped until the temperature parameter drops to a threshold, and the candidate node with the highest probability amplitude in the quantum superposition state when the iterative update is stopped is selected as the optimal mapping node; Based on the optimal mapping node for each field in the intermediate format data, combined with the semantic constraints and standard format of the nodes in the knowledge graph, the data corresponding to each field in the intermediate format data is standardized and mapped to obtain standardized data.

2. The method according to claim 1, wherein The construction process of the knowledge graph includes: Determine the core entities and attributes of the external multi-source data as nodes of the initial knowledge graph, wherein the nodes include entity nodes and attribute nodes; Based on each sample field in the external multi-source data, determining a mapping node for the sample field using the semantic similarity between the sample field and each of the nodes, and mapping the sample field to the initial knowledge graph based on the mapping node; In the initial knowledge graph after mapping and filling, the semantic constraints and standard formats of the attribute nodes are set to obtain a knowledge graph for data conversion.

3. The method according to claim 1, wherein The process of calculating the semantic similarity between the field and each node in the knowledge graph includes: Extracting semantic vectors of the fields using named entity recognition technology; By calculating the cosine similarity between the semantic vector of the field and the semantic vector of each node in the knowledge graph, the semantic similarity between the field and each of the nodes is obtained.

4. The method according to claim 1, wherein The method of converting the original data into intermediate format data by using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm includes: Identifying the format of the original data using a dynamic format adapter and parsing the initial field type in the original data; Converting the original data into an internal data structure using a parser based on the format, recorded as initial parsed data; Correcting the initial field type based on the initial parsed data using a reinforcement learning schema algorithm to obtain the field type in the original data; Based on the data conversion mapping rules and the machine learning algorithm, the original data is converted into intermediate format data in combination with the field type.

5. The method according to claim 4, wherein The converting of the original data into intermediate format data based on the data conversion mapping rules and the machine learning algorithm in combination with the field type includes: Determining, based on the field type of the original data and the predefined mapping rule, original data for which the mapping rule exists and original data for which the mapping rule does not exist; Based on the field type of the original data and the mapping rule, converting the original data with the mapping rule into intermediate format data; A machine learning algorithm is used to perform semantic inference and conversion on the original data for which no mapping rules exist based on the field type of the original data to obtain intermediate format data.

6. The method according to claim 1, wherein Before converting the original data into intermediate format data using the dynamic format adapter based on the data conversion mapping rules and the machine learning algorithm, the method further includes: Edge computing technology is used to extract and filter the raw data from the multiple different data sources to obtain processed raw data.

7. A big data standardization system based on data integration, characterized in that: include: Data acquisition module, used to obtain raw data from multiple different data sources; An intermediate format conversion module, configured to convert the original data into intermediate format data using a dynamic format adapter based on data conversion mapping rules combined with a reinforcement learning Schema algorithm; A node solving module is used to generate a quantum superposition state of each field in the intermediate format data using quantum state coding and a node in the knowledge graph, including: selecting multiple candidate mapping nodes for the field from the knowledge graph based on the semantic similarity between the field and each node in the knowledge graph, combined with a similarity threshold; quantizing the semantic relationship between the field and each candidate mapping node into a probability amplitude using quantum state coding; and obtaining a quantum superposition state of the field and the node in the knowledge graph by combining the probability amplitude between the field and each candidate mapping node; The node solving module is further configured to solve the optimal mapping node for the field in the knowledge graph based on the quantum superposition state using a quantum annealing algorithm, including: utilizing the quantum tunneling effect in the quantum annealing algorithm, based on the quantum superposition state, preset temperature parameters, and energy parameters, iteratively updating the probability amplitude of the candidate mapping node in the quantum superposition state multiple times; after each iterative update, actively reducing the temperature parameter according to a preset attenuation coefficient; stopping the iterative update until the temperature parameter drops to a threshold, and selecting the candidate node with the highest probability amplitude in the quantum superposition state when the iterative update is stopped as the optimal mapping node; A standardized mapping module is used to perform standardized mapping on the data corresponding to each field in the intermediate format data based on the optimal mapping node for each field in the intermediate format data, combined with the semantic constraints and standard format of the nodes in the knowledge graph, to obtain standardized data.

8. The system according to claim 7, wherein: The node solving module is specifically used for: Determine the core entities and attributes of the external multi-source data as nodes of the initial knowledge graph, wherein the nodes include entity nodes and attribute nodes; Based on each sample field in the external multi-source data, determining a mapping node for the sample field using the semantic similarity between the sample field and each of the nodes, and mapping the sample field to the initial knowledge graph based on the mapping node; In the initial knowledge graph after mapping and filling, the semantic constraints and standard formats of the attribute nodes are set to obtain a knowledge graph for data conversion.

Images