Multi-source heterogeneous data integration method, system and device and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a multi-source heterogeneous data integration method, system and device and a storage medium, and the method comprises the steps: obtaining multi-source heterogeneous data, and generating an identification attribute for each piece of heterogeneous data; dividing the heterogeneous data with the same identification attribute into the same data group; converting the heterogeneous data in the data group into standard data in a specified format; based on a predefined integration rule, integrating the plurality of pieces of standard data converted from the data set into one piece of complete data, the integration rule comprising a conflict resolution strategy and / or a data association rule; and storing the complete data to a corresponding position of a database according to a predefined storage strategy. According to the method, effective integration of multi-source heterogeneous data is realized, and the data analysis efficiency and quality are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a multi-source heterogeneous data integration method, system, device and storage medium. BACKGROUND

[0002] In the scenarios of grassroots governance, government service and enterprise operation, data sources are extensive and formats are various, including: structured data such as relational database tables and Excel tables; semi-structured data such as JSON and XML files; and unstructured data such as scanned copies, pictures and text files.

[0003] The traditional integration method converts multi-source data into a unified format, but has the following essential defects: cross-source data fragments of the same object cannot be associated, resulting in information fragmentation and incomplete decision basis during data analysis. SUMMARY

[0004] In view of the above problems of the prior art, the application provides a multi-source heterogeneous data integration method, system, device and storage medium to solve the above technical problems.

[0005] In a first aspect, the application provides a multi-source heterogeneous data integration method, comprising: obtaining multi-source heterogeneous data and generating an identification attribute for each piece of heterogeneous data; dividing heterogeneous data with the same identification attribute into the same data group; converting the heterogeneous data in the data group into standard data in a specified format; integrating the multiple pieces of standard data converted by the data group into one complete data based on a predefined integration rule, the integration rule including a conflict resolution strategy and / or a data association rule; storing the complete data to a corresponding position in a database according to a predefined storage strategy.

[0006] In an optional embodiment, obtaining multi-source heterogeneous data and generating an identification attribute for each piece of heterogeneous data comprises: obtaining multi-source heterogeneous data through a multi-source interface platform; generating an identification attribute for the heterogeneous data based on the object to which the heterogeneous data belongs.

[0007] In an optional embodiment, generating an identification attribute for the heterogeneous data based on the object to which the heterogeneous data belongs comprises: analyzing the object to which the heterogeneous data belongs, the information of the object including one or more of a user ID, a device serial number, an order number and a geographic location code; performing field normalization processing on the object, and constructing a structured combination key based on the normalized processing and the object; generating a hash digest of the structured combination key and setting the hash digest as an identification attribute of the heterogeneous data.

[0008] In an optional embodiment, the heterogeneous data in the data group is converted into standard data in a specified format, including: calling a predefined conversion rule library, the conversion rule library containing field mapping relationships and format conversion functions corresponding to data group types; parsing the original fields of each piece of heterogeneous data in the data group; mapping the original fields to target standard fields according to the field mapping relationships; performing data type conversion, unit standardization, or structure reconstruction on the mapped field values through the format conversion functions; outputting standard data conforming to the specified format.

[0009] In an optional embodiment, the conflict resolution strategy includes: identifying standard data with different values for the same target field as conflict data; performing at least one of the following operations on the conflict data: covering standard data with low data source priority with standard data with high data source priority; selecting the latest value according to the timestamp; calculating the weighted average value for numerical conflict fields; triggering a manual review process and recording conflict logs.

[0010] In an optional embodiment, based on predefined integration rules, the converted multiple pieces of standard data in the data group are integrated into one complete data, including: conflict resolution on multiple pieces of standard data in the data group; aggregating time series data in the data group according to time windows, including: setting sliding time window parameters and performing at least one of the following operations on the numerical fields of time series data in the time window: calculating statistics, extracting key sampling points, and applying interpolation algorithms to fill in missing values before aggregation; outputting the aggregation result as the target time series field value; performing logical splicing on complementary fields, including: identifying complementary field pairs in the data group that have logical associations, the complementary field pairs including discrete field combinations, spatial coordinates, and segmented text combinations; reconstructing fields according to predefined splicing templates, including: performing string concatenation on discrete fields; generating GeoJSON structures for spatial coordinates; and sequentially splicing segmented texts after adding separators.

[0011] In an optional embodiment, the complete data is stored to a corresponding position of a database according to a predefined storage strategy, including: determining the storage position according to the identification attribute corresponding to the data group; writing the complete data of the data group to the storage position.

[0012] In a second aspect, the application provides a multi-source heterogeneous data integration system, including: an attribute generation module configured to acquire multi-source heterogeneous data and generate an identification attribute for each piece of heterogeneous data; a data grouping module configured to group heterogeneous data with the same identification attribute into a same data group; a format conversion module configured to convert the heterogeneous data in the data group into standard data in a specified format; a data integration module configured to integrate the multiple pieces of standard data converted from the data group into a complete data based on a predefined integration rule, the integration rule including a conflict resolution strategy and / or a data association rule; a data storage module configured to store the complete data to a corresponding position of a database according to a predefined storage strategy.

[0013] In a third aspect, a device is provided, including: a memory configured to store an integration program of multi-source heterogeneous data; a processor configured to implement the steps of the integration method of multi-source heterogeneous data when executing the integration program of multi-source heterogeneous data.

[0014] In a fourth aspect, a computer readable storage medium is provided, and the storage medium stores an integration program of multi-source heterogeneous data, and the integration program of multi-source heterogeneous data implements the steps of the integration method of multi-source heterogeneous data when executed by a processor.

[0015] The application has the advantages that the integration method, system, device and storage medium of multi-source heterogeneous data provided by the application automatically match the scattered data of the same object through the identification attribute, construct a multi-dimensional panoramic portrait, and solve the problem of information fragmentation; automatically process multi-source conflicts (such as enterprise inventory difference values) based on a predefined strategy (priority / time stamp / aggregation calculation), guarantee the reliability of data decision-making; replace manual matching with logical splicing rules (such as automatically generating complete archives of the masses), and improve the integration efficiency; support real-time analysis after storing the complete data, and improve the data analysis quality and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0017] Figure 1 is a schematic flow chart of a method of one embodiment of the present application.

[0018] Figure 2 is a schematic block diagram of a system of one embodiment of the present application.

[0019] Figure 3 is a structural schematic diagram of an apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the technical personnel in the art better understand the technical solutions in the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort should belong to the scope of protection of the present application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0022] The integration method of multi-source heterogeneous data provided by the embodiments of the present application is executed by a computer device, and accordingly, the integration system of multi-source heterogeneous data runs in the computer device.

[0023] Figure 1 is a schematic flow chart of a method of one embodiment of the present application. Wherein, Figure 1 The execution subject can be an integration system of multi-source heterogeneous data. According to different needs, the order of steps in the flow chart can be changed, and some can be omitted.

[0024] As Figure 1 shown, the method comprises: S1. acquiring multi-source heterogeneous data, and generating an identification attribute for each piece of heterogeneous data; S2. dividing the heterogeneous data with the same identification attribute into the same data group; S3. converting the heterogeneous data in the data group into standard data of a specified format; S4. Integrating the converted multiple pieces of standard data of the data set into one complete data based on predefined integration rules, the integration rules including conflict resolution strategies and / or data association rules; S5. Storing the complete data into a corresponding position of a database according to a predefined storage strategy.

[0025] In an embodiment of the present application, based on step S1, a possible embodiment will be given below to specifically and non-limitingly illustrate the implementation thereof.

[0026] S101. Obtaining multi-source heterogeneous data through a multi-source interface platform.

[0027] To achieve efficient aggregation of multi-source heterogeneous data, the multi-source interface platform needs to build a hierarchical interface adaptation architecture, covering a protocol adaptation layer, a data conversion layer, and an access management layer. The protocol adaptation layer dynamically adapts the interface protocols of different data sources through a pre-installed protocol analysis engine (supporting HTTP / HTTPS, MQTT, Kafka, FTP, etc. mainstream protocols), for example, the MQTT protocol for Internet of Things devices, which uses an asynchronous subscription mechanism to achieve real-time data pushing, while the HTTP interface for enterprise ERP systems uses a timing task to trigger batch data pulling. The data conversion layer introduces a common data model (CDM) as an intermediate carrier, which uniformly maps structured data (such as relational database tables), semi-structured data (such as JSON, XML files), and unstructured data (such as text, images, audio) into a standardized format containing field name, data type, length, and constraint conditions. For unstructured data, descriptive fields are constructed by extracting file metadata (such as creation time, file size, format type) to achieve compatibility with CDM. The access management layer uses a distributed task scheduling framework (such as Apache Airflow) to dynamically allocate computing resources based on the update frequency, data size, and priority of the data source, while preventing the overall platform from collapsing due to a single data source failure through a fuse mechanism (such as Resilience4j). For high-concurrency scenarios, a message queue (such as RabbitMQ) is introduced as a buffer layer to ensure the stability of data access and improve the success rate of data access through traffic peak clipping and staggered processing.

[0028] S102. Generating an identification attribute for the heterogeneous data based on the object to which the heterogeneous data belongs.

[0029] (1) Analyzing the object to which the heterogeneous data belongs, the information of the object including one or more of user ID, device serial number, order number, and geographic location code.

[0030] For the belonging object information of heterogeneous data, a multi-dimensional object feature extractor is designed to achieve accurate analysis through the collaborative work of rule engine and machine learning model. The rule engine extracts structured identifiers such as user ID (e.g. user_\\d{10} format), device serial number (e.g. SN-\\w{8}-\\d{4} format), order number (e.g. ORD-\\d{8}-\\w{6} format) from data fields based on pre-set semantic matching rules (such as regular expressions); for unstructured data (such as text logs), a named entity recognition (NER) model (such as a BERT fine-tuning model) is used to identify implicit object information such as geographic location codes (such as latitude and longitude coordinates, administrative division codes).

[0031] (2) Field normalization processing is performed on the belonging object, and a structured combination key is constructed based on the normalized processing and the belonging object.

[0032] A field standardization mapping library is constructed to address the same name different meaning or different name same meaning problem of different data sources through the ontology method to establish a unified field semantic system. For example, fields such as "user number", "user_no", "U_ID" are standardized to "user_id"; for geographic location codes, format conversion is performed according to the standard coding rules of the national geographic information public service platform (such as GB / T 2260 administrative division code) to ensure consistency of the same object identifier in different data sources. For missing values and outliers, a hierarchical processing strategy is adopted: interpolation method is used for numerical type identifier (such as the number field in device serial number), and pattern matching is used to generate substitute values for character type identifier (such as the letter segment in user ID); for abnormal identifiers that obviously deviate from the rules (such as length inconsistency, format error), manual verification process is triggered to avoid invalid data entering the subsequent link.

[0033] (3) A hash digest of the structured combination key is generated, and the hash digest is set as the identification attribute of the heterogeneous data.

[0034] The structured combination key adopts a hierarchical field splicing strategy, and the priority is sorted according to "core identifier - auxiliary identifier - timestamp", for example, user_id|device_sn|order_no|geo_code|timestamp, where the timestamp is accurate to the millisecond level, used to distinguish data generated at different times for the same object. A separator checking mechanism is introduced in the splicing process to avoid key value ambiguity caused by field content conflict through special character (such as "|") escape processing. Hash digest generation uses a cryptographic-grade hash algorithm (such as SHA-256). Structured key combinations are standardized in UTF-8 encoding before digest generation to ensure character set consistency. To improve hash calculation efficiency, a pre-calculated cache mechanism is introduced, creating a hash value mapping table for frequently occurring key combinations. This strategy trades space for time to keep the time required for a single hash calculation under 1ms. The resulting hash digest (256-bit binary value) serves as a unique identifier for heterogeneous data and is embedded in the metadata area of ​​the data header, supporting subsequent data association, deduplication, and traceability.

[0035] In an embodiment of the present invention, based on step S3, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0036] S301. Build a rule base.

[0037] The transformation rule base adopts a multi-level knowledge graph architecture and implements dynamic management and call of rules through metadata-driven approach. The core of the rule base consists of domain ontology layer, mapping rule layer and execution engine layer: Domain ontology layer: Builds a data semantic model based on OWL (Web Ontology Language), defines data group types (such as user data, transaction data, and device status data) and the entity relationships they contain, and implements hierarchical organization of data types through a conceptual classification system (Taxonomy).

[0038] Mapping rule layer: Use XML Schema to define field mapping relationships and implement path addressing of heterogeneous data sources through XPath expressions. For example, map the NAME field of the relational database table USER_INFO to the full_name field of the standard model. Execution engine layer: This layer compiles and executes format conversion functions based on a regular expression language (such as Drools). It supports basic functions such as arithmetic operations, string processing, and date formatting, as well as complex machine learning model calls (such as data cleaning and anomaly detection).

[0039] S302. Heterogeneous data parsing and mapping mechanism, using an adaptive parsing framework to achieve intelligent identification and processing of different data structures: Data structure detection: Identify data structural features such as delimiter type (comma, tab), nesting level, key-value pair format, etc. through sampling analysis, and automatically select the optimal parser (such as CSVParser, JSONParser, XMLParser).

[0040] Pattern matching algorithm: based on regular expressions and finite state automata (FSA) to realize the semantic recognition of original fields; Dynamic mapping strategy: for complex scenarios (such as one-to-many, many-to-one mapping), a context-aware mapping decision tree is used to select the most suitable mapping rule according to the business context of the data (such as source system, data theme), and the mapping accuracy rate reaches more than 98.5%.

[0041] The format conversion process adopts a pipeline processing architecture, and function extension is realized through plug-in conversion components: Data type conversion: based on type system (such as Avro Schema) to realize cross-language type mapping, for example, convert Java's java.util.Date type to standard ISO 8601 time format; Unit standardization: build a physical quantity unit conversion knowledge base (such as length, weight, currency unit), and realize automatic conversion through dimension analysis. For example, convert imperial units to metric units; Structural reconstruction: use JSONPath and XSLT technology to realize the flattening or nesting conversion of data structure, for example, convert hierarchical JSON objects to relational table structure.

[0042] Establish a multi-dimensional quality evaluation system, and realize quality control in the standardization process through data verification rules (such as value range constraint, uniqueness constraint, correlation integrity constraint): Exception handling strategy: adopt hierarchical processing mechanism, perform automatic repair for repairable exceptions (such as format error), record audit log and trigger manual intervention process for unrepairable exceptions (such as missing key field), and ensure the robustness of data standardization process.

[0043] In one embodiment of the present application, based on step S4, a possible embodiment will be given below to non-restrictively explain the specific implementation scheme.

[0044] S401. Conflict resolution is performed on multiple standard data in the data set.

[0045] The conflict resolution process is built on a multi-level conflict identification framework, and the precise positioning of conflict data is realized through semantic similarity calculation and feature matching. For multi-value differences of the same target field, first, a pre-defined data source credibility evaluation model is used for priority sorting, which takes into account the historical accuracy, update frequency and data integrity of the data source to form a credibility weight value in the 0-1 interval. When the weight difference is more than 0.3, the high-priority coverage mechanism is triggered. The timestamp screening mechanism adopts time granularity normalization processing to convert different formats of time identifiers into Unix timestamps, judges data freshness by calculating the time difference threshold (default setting is 1.5 times of the data acquisition period), and ensures the accuracy of the latest value selection. For numerical conflict fields, a weighted average value calculation introduces a dynamic weight allocation algorithm. The weight factor is positively correlated with the data source reliability and the data sample size, and satisfies the constraint condition that the sum of the weights is 1. The calculation formula is:

[0046] α and β are adjustment coefficients, c i is the data source reliability, s i is the sample size.

[0047] The artificial review process embeds a conflict severity grading mechanism. When the conflict field involves a key business indicator (such as transaction amount, user identity information) or the difference value exceeds the preset threshold (relative error of numerical fields > 10%, edit distance of character fields > 3), a three-level review process is automatically triggered, and a structured conflict log containing the original value of the conflict field, the basis for resolution, and the handler information is generated. The log uses blockchain storage technology to ensure tamper resistance.

[0048] S402. Aggregate the time series data in the data set according to the time window, including: setting the sliding time window parameters, and performing at least one of the following operations on the numerical fields of the time series data within the time window: calculating statistics, extracting key sampling points, and filling missing values with interpolation algorithms before aggregation; output the aggregation result as the target time series field value.

[0049] Time series data aggregation is based on an adaptive sliding window model. The window parameters are dynamically adjusted by two factors: the base window length is set according to the data acquisition frequency (such as 5 minutes for second-level data), and the sliding step is 1 / 5-1 / 3 of the window length, to balance data granularity and computational efficiency. The boundary calibration of the time window uses a time alignment algorithm to map non-equidistant sampling data to a unified time axis through linear interpolation, eliminating time deviation caused by clock drift. Statistical quantity calculation includes descriptive statistics (mean, variance, extreme value) and high-order statistical characteristics (skewness, kurtosis). For high-frequency fluctuation data (such as real-time sensor monitoring values), robust statistical quantities (such as median, interquartile range) are introduced to suppress the interference of outliers. Key sampling point extraction uses a curvature-based feature point detection algorithm. By calculating the absolute value of the second derivative of the data sequence, points with curvature exceeding the threshold (dynamically adjusted based on data fluctuation variance) are marked as key feature points, and the retention ratio is controlled at 10%-30% of the original data volume. The missing value filling adopts a multi-scale interpolation strategy: missing values with a time interval less than twice the sampling period are filled by linear interpolation; longer missing sections are filled by combining forward filling and trend extrapolation, by fitting the trend item of the data sequence (fitting a polynomial curve by least squares method) to achieve reasonable filling, ensuring the time continuity and numerical rationality of the aggregation results.

[0050] S403. Performing logical splicing on the complementary fields, including: identifying complementary field pairs with logical association within the data group, the complementary field pairs including discrete field combinations, spatial coordinates, segmented text combinations; reconstructing the fields according to a predefined splicing template, including: performing string connection on the discrete fields; generating a GeoJSON structure for the spatial coordinates; and sequentially splicing the segmented texts after adding a separator.

[0051] The complementary field identification relies on a semantic correlation calculation model to mine the logical relationship between fields by constructing a domain knowledge graph. For discrete field combinations, the association is determined based on a co-occurrence probability threshold (set to 0.85); the spatial coordinates field is identified by calculating the spatial clustering coefficient of the coordinate point set (using the DBSCAN algorithm) to identify geographical association; and the segmented text combinations use the BERT model to calculate the semantic similarity between text segments, and a similarity score of more than 0.7 is considered as an effective association. The field reconstruction process adopts a template-driven structured conversion mechanism: the discrete field splicing introduces semantic separators (such as "|", ";"), and the separator selection is dynamically adjusted according to the field type to avoid ambiguity; when converting spatial coordinates into a GeoJSON structure, the topology relationship description includes point, line, and face types, and the coordinate precision is retained to 6 decimal places (about 10 cm spatial resolution); the segmented text splicing uses a context-aware separator insertion strategy to add line breaks or paragraph markers at semantic turning points (identified by syntactic semantic analysis), ensuring text coherence. The splicing template supports dynamic parameter configuration, and the field mapping rules and format constraints are defined by a preset XML format template, which includes field order, data type verification rules, and output format description. Different business scenarios can be flexibly adapted through template version control.

[0052] In one embodiment of the present application, based on step S5, a possible embodiment will be given below to non-restrictively illustrate the specific implementation scheme.

[0053] S501. Storage location mapping strategy based on identified attributes A multi-level storage indexing system is constructed to realize intelligent positioning of data through the hash features of identified attributes, and the core is to establish a precise mapping relationship between identified attributes and physical storage nodes. The consistent hashing algorithm is used to uniformly divide the hash space of the identification attribute into 2^160 virtual nodes, and each physical storage node manages the continuous hash interval through the virtual node proxy. Specifically, the SHA-1 operation is performed on the hash digest of the identification attribute to generate a 160-bit hash value, the mapping relationship between the hash value and the virtual node is determined, and the final positioning is completed through the preset virtual node-physical node routing table. This mechanism can control the data migration amount within 10% of the change in the number of nodes when the nodes are dynamically added or reduced, significantly reducing the impact of cluster expansion and contraction on service availability. A composite partition key strategy is designed to implement multi-level storage partitioning: the first-level partition divides independent storage clusters based on the business type of the data source (such as user behavior data, transaction flow data, and Internet of Things sensor data), ensuring data isolation; the second-level partition uses time granularity layering (three-level nesting by year-month-day), and time-series data is classified into corresponding partitions according to the generation time, laying the foundation for hot and cold data separation; the third-level partition uses modulo operation (with a modulus value of 1.5 times the number of physical nodes) on the hash value of the identification attribute to achieve data sharding, avoiding excessive load on a single node. A dynamic matching model of identification attribute-storage engine is established to select the optimal storage scheme based on the access characteristics of the data: for high write-throughput scenarios (write / read ratio > 10:1), match the LSM tree structure storage engine to improve write performance; for complex aggregation query scenarios (multi-field joint query ratio > 70%), adapt the columnar storage engine to optimize query efficiency; for time-series data with significant time characteristics (time dimension filter condition frequency > 80%), use a time-series database to reduce storage redundancy; for semi-structured data (nesting level > 3 layers), use a document-type database to preserve its native structure. S502. Distributed storage architecture design A lake-warehouse integrated architecture is used to implement data lifecycle management, and the architecture is divided into three storage layers: the raw data layer is based on a distributed object storage system (such as MinIO) to store complete data groups, using Parquet columnar storage format compression with a compression ratio of 1:8 to 1:10; the structured processing layer introduces transactional data lake technology (such as DeltaLake), supporting ACID features and time travel queries to ensure atomicity of data updates; the data service layer deploys an MPP architecture data warehouse (such as StarRocks) to accelerate queries through pre-computed materialized views and global indexes, controlling complex analysis query response time to seconds. The metadata management adopts a graph structure model, and records data blood relationship through a graph database (such as Neo4j): the node types include data sources, data groups, storage locations, and processing tasks, and the edge types are defined as generation relationship, storage relationship, and conversion relationship; a metadata full-text index is constructed based on Lucene, millisecond-level retrieval is supported by identifying attributes, and the metadata update delay is not more than 500 ms. S503. Data writing optimization strategy A high-performance writing pipeline is constructed, and the writing efficiency is improved through batch processing optimization: a micro-batch processing mechanism is adopted, batch processing is triggered according to a 5-second time window and a 100MB data volume threshold, data in the batch is sorted according to three-level partition keys and then written, and small file generation is reduced; multi-partition parallel writing is implemented through a thread pool, and the parallel degree is dynamically adjusted according to the CPU utilization of the cluster (the highest is 2 times the number of nodes when idle, and is reduced to 50% of the number of nodes when busy); a two-phase commit protocol is adopted to ensure the atomicity of batch processing and improve the success rate of submission. The compression encoding strategy adopts a type adaptive mechanism: ZSTD compression algorithm is selected for text data (compression speed > 300MB / s, compression ratio 1:3.5); numerical data matches LZ4 compression algorithm (decompression speed > 1GB / s); and Snappy compression is adopted for binary data to balance performance and efficiency. Error handling implements idempotent writing: data uniqueness is quickly checked through a Bloom filter before writing; conflict data is handled according to the principle of 'latest writing priority', and the latest version is retained; the exponential backoff retry mechanism is adopted for failed tasks (retry interval 100ms→200ms→400ms→800ms→1600ms), and the maximum number of retries is set to 5, to ensure eventual consistency.

[0054] In some embodiments, the multi-source heterogeneous data integration system can include a plurality of functional modules composed of computer program segments. The computer programs of each program segment in the multi-source heterogeneous data integration system can be stored in the memory of the computer device and executed by at least one processor to perform the functions of integrating multi-source heterogeneous data (see Figure 1 Description).

[0055] In this embodiment, the multi-source heterogeneous data integration system can be divided into a plurality of functional modules according to the functions it performs, as shown in Figure 2 The module referred to in the present application refers to a series of computer program segments that can be executed by at least one processor and can complete a fixed function, which are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0056] The attribute generation module is configured to obtain multi-source heterogeneous data and generate an identification attribute for each piece of heterogeneous data. a data grouping module, configured to group heterogeneous data with the same identification attribute into a same data group; a format conversion module, configured to convert the heterogeneous data in the data group into standard data in a specified format; a data integration module, configured to integrate the converted standard data in the data group into a complete data based on a predefined integration rule, the integration rule including a conflict resolution strategy and / or a data association rule; a data storage module, configured to store the complete data into a corresponding position of a database according to a predefined storage strategy.

[0057] Figure 3 The method for integrating multi-source heterogeneous data provided by the embodiments of the present application can be applied to a device. Those skilled in the art can understand that the device structure involved in the embodiments of the present application does not constitute a limitation on the device, and the device can include more or fewer components than the diagram, or combine certain components, or different component arrangements. In the embodiments of the present application, the device includes but is not limited to a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described herein and / or claimed.

[0058] The device 300 can include a processor 310, a memory 320, and a communication unit 330. These components communicate through one or more buses, and those skilled in the art can understand that the structure of the server shown in the diagram does not constitute a limitation on the present application, and it can be a bus structure or a star structure, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0059] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 can execute part or all of the steps in the following method embodiments.

[0060] The processor 310 is the control center of the storage device, connects various parts of the entire electronic device by using various interfaces and lines, and performs various functions of the electronic device and / or processes data by running or executing software programs and / or modules stored in the memory 320 and calling data stored in the memory. The processor can be composed of an integrated circuit (IC), for example, can be composed of a single packaged IC, or can be composed of a plurality of packaged ICs connected together. For example, the processor 310 can only include a central processing unit (CPU). In the embodiments of the present application, the CPU can be a single operation core or can include multiple operation cores.

[0061] The communication unit 330 is configured to establish a communication channel, so that the storage device can communicate with other devices. The communication unit 330 receives user data sent by other devices or sends user data to other devices.

[0062] The present application also provides a computer storage medium, wherein the computer storage medium can store a program, and the program can include some or all steps in the embodiments of the present application when executed. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0063] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. The software product includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0064] In the present specification, the same or similar parts among various embodiments can be referred to each other. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0065] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. For example, the system embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different modules can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0066] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed on a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0067] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0068] Although the present application has been described in detail by referring to the preferred embodiments thereof, it is to be understood that the present application is not limited to the embodiments described above. Rather, it should be appreciated that those skilled in the art, upon consideration of the disclosure, can make modifications and / or improvements to the embodiments of the application without deviating from the spirit and scope of the application. Any modifications and / or improvements made to the embodiments of the application by those skilled in the art are to be considered within the scope of the application.

Claims

1. A method for integrating multi-source heterogeneous data, characterized in that, The method comprises the following steps: acquiring multi-source heterogeneous data and generating an identification attribute for each piece of heterogeneous data; dividing heterogeneous data with the same identification attribute into the same data group; converting the heterogeneous data in the data group into standard data in a specified format; integrating the converted multiple pieces of standard data in the data group into one complete data based on a predefined integration rule, wherein the integration rule comprises a conflict resolution strategy and / or a data association rule; storing the complete data in a corresponding position of a database according to a predefined storage strategy.

2. The method of claim 1, wherein, The method comprises the following steps: acquiring multi-source heterogeneous data and generating an identification attribute for each piece of heterogeneous data; acquiring multi-source heterogeneous data through a multi-source interface platform; 3. The method of claim 1, wherein, generating an identification attribute for the heterogeneous data based on the object to which the heterogeneous data belongs. The method comprises the following steps: analyzing the object to which the heterogeneous data belongs, wherein the information of the object comprises one or more of a user ID, a device serial number, an order number and a geographic location code; performing field normalization processing on the object, and constructing a structured combination key based on the normalized object; 4. The method of claim 1, wherein, generating a hash digest of the structured combination key, and setting the hash digest as the identification attribute of the heterogeneous data. The method comprises the following steps: calling a predefined conversion rule library, wherein the conversion rule library comprises a field mapping relationship and a format conversion function corresponding to the type of the data group; analyzing the original field of each piece of heterogeneous data in the data group; mapping the original field to a target standard field according to the field mapping relationship; performing data type conversion, unit standardization or structure reconstruction on the mapped field value through the format conversion function; 5. The method of claim 1, wherein, outputting the standard data conforming to the specified format. The conflict resolution strategy comprises: identifying standard data with different values in the same target field as conflict data; performing at least one of the following operations on the conflict data: covering standard data with a low data source priority with standard data with a high data source priority; selecting the latest value according to the timestamp; calculating the weighted average value of the numerical conflict field; 6. The method of claim 1, wherein, triggering a manual review process and recording a conflict log. The method comprises the following steps: resolving conflicts of the multiple pieces of standard data in the data group; aggregating time series data in the data group according to a time window, comprising: setting a sliding time window parameter, and performing at least one of the following operations on the numerical field of the time series data in the time window: calculating a statistical quantity, extracting a key sampling point, and filling in missing values by applying an interpolation algorithm and then aggregating; outputting the aggregation result as the target time series field value; performing logical splicing on complementary fields, comprising:

7. The method of claim 1, wherein, identifying a complementary field pair with logical association in the data group, wherein the complementary field pair comprises a discrete field combination, a spatial coordinate and a segmented text combination; reconstructing the fields according to a predefined splicing template, comprising: performing string concatenation on the discrete fields; generating a GeoJSON structure for the spatial coordinate; and sequentially splicing the segmented text after adding a separator. The method comprises the following steps: determining a storage location according to an identification attribute corresponding to the data group; writing complete data of the data group into the storage location.

8. A system for integrating multi-source heterogeneous data, characterized in that, The method comprises the steps of: an attribute generation module configured to acquire multi-source heterogeneous data and generate an identification attribute for each piece of heterogeneous data; a data grouping module configured to group heterogeneous data with the same identification attribute into the same data group; a format conversion module configured to convert the heterogeneous data in the data group into standard data in a specified format; a data integration module configured to integrate the converted standard data in the data group into complete data based on a predefined integration rule, the integration rule comprising a conflict resolution strategy and / or a data association rule; a data storage module configured to store the complete data into a corresponding location of a database according to a predefined storage strategy.

9. A device for integrating multi-source heterogeneous data, characterized in that, The method comprises the steps of: a memory configured to store an integration program of multi-source heterogeneous data; a processor configured to execute the integration program of multi-source heterogeneous data to implement the steps of the integration method of multi-source heterogeneous data according to any one of claims 1-7.

10. A computer readable storage medium storing a computer program, characterized in that, The readable storage medium stores an integration program of multi-source heterogeneous data, and the integration program of multi-source heterogeneous data is executed by a processor to implement the steps of the integration method of multi-source heterogeneous data according to any one of claims 1-7.

Images