Method for integrating and analyzing heterogeneous source data

Abstract

The invention discloses a method for integrating and analyzing heterogeneous data (3) from a heterogeneous source application (2). The method is used for solving the inherent non-standardization problem of a data model used by a Web-based database platform. All platforms adopt different data configurations, so that a unified and standard data presentation mode cannot be followed, and structural aggregation of information is not facilitated. In order to achieve the purpose, the computer implementation method comprises the steps that a calculation data collection program (1) is executed to access and load heterogeneous source data (3); providing a heterogeneous data integration module (4) for converting the heterogeneous source data (3) into target data (5); performing a data analysis process (6) based at least on the target data (5) to generate output data (7); and writing the output data (7) to the normalized database (8).

Description

Technical Field

[0001] This application relates to the technical field of integrating data from heterogeneous source applications, and more specifically, to a method for integrating and analyzing heterogeneous data from heterogeneous source applications. Background Technology

[0002] Heterogeneous data refers to data that exhibits high diversity in data type and format. Data from different source applications is typically heterogeneous because it originates from independent and decentralized activities and is managed and maintained by different application owners. Therefore, even if these data relate to the same phenomenon, their data values ​​and data structures often differ from one another.

[0003] In data integration systems, effectively and efficiently handling data heterogeneity is crucial and a prerequisite for realizing various other applications. For example, in a "data web" environment, the ability to handle data heterogeneity enables semantic retrieval of entities and their relationships based on the text web, and deep reasoning using relevant ontologies, thereby constructing a "knowledge web".

[0004] Therefore, when dealing with applications from multiple heterogeneous data sources, the ultimate goal is usually to merge different representations of the same real-world entity to obtain a unified view, making the user feel as if they are interacting with a single data source.

[0005] Currently, databases used for design and engineering employ a variety of different data models, interface languages, naming conventions, data semantics, schemas, and data representations. Therefore, sharing heterogeneous information among multiple design resources is a fundamental problem in concurrent engineering. Successful concurrent engineering also requires access to data across multiple stages of the design lifecycle, but the differences in data across different tools and stages present significant obstacles.

[0006] The solution proposed in this application aims to overcome the above-mentioned problems in an innovative way. Summary of the Invention

[0007] Therefore, the purpose of this application is to provide a method for integrating and analyzing heterogeneous source data, in order to solve the inherent non-standardization problem of the data models used by different Web-based database platforms. Each Web-based database platform follows its own data configuration and therefore does not adopt a unified and standardized data presentation method, which is not conducive to the structured aggregation of information.

[0008] To achieve the above objectives, this application provides a computer-implemented method for integrating and analyzing heterogeneous source data, used to process heterogeneous source data obtained from multiple heterogeneous source applications. The method includes the following steps: Execute a computational data acquisition procedure to access and load heterogeneous source data from at least one heterogeneous source application; Provide specifications for the heterogeneous data integration module to convert heterogeneous source data into target data; At least perform data analysis and processing based on the target data to generate output data; Write the output data into a normalized database.

[0009] More specifically, the specifications for heterogeneous data integration modules include: Input the first-level data structure specification, which describes the intermediate representation of the data, and the second-level data structure specification, which describes the target data representation; Implement an extraction framework specification to extract data from heterogeneous source data to generate an intermediate representation of the data; Implement a data normalization framework to transform intermediate representations of data into target data.

[0010] The proposed solution utilizes and co-orchestrates a set of innovative processes running synchronously, such as those based on robotic process automation, natural language processing, and artificial intelligence mechanisms, to collect data from various heterogeneous source applications, process and analyze the collected data, and perform a set of advanced data analyses to provide users with integrated information.

[0011] The innovative nature of the method described in this application lies in the synchronous and automated operation of the aforementioned elements for the purpose of generating knowledge; at the same time, it also constitutes a technical challenge in establishing an efficient and effective interactive relationship among all elements. Attached Figure Description

[0012] Figure 1 is a block diagram illustrating a first exemplary data element chain according to the method described in this application. Reference numerals are: 1 — Computational data acquisition program; 2 — Application of heterogeneous sources; 3 — Heterogeneous source data; 4 — Heterogeneous data integration module; 5 — Target data; 6 — Data analysis and processing; 7 — Output data; 8 — Standardized database.

[0013] Figure 2 is a block diagram illustrating a second exemplary data element chain according to the method described in this application. Reference numerals are: 1 — Computational data acquisition program; 2 — Application of heterogeneous sources; 3 — Heterogeneous source data; 4 — Heterogeneous data integration module; 5 — Target data; 6 — Data analysis and processing; 7 — Output data; 8 — Standardized database; 9 — User interface platform; 10 — User profile data. Detailed Implementation

[0014] A more general and advantageous configuration of the method described herein is shown in the Summary of the Invention section of this application. Such a configuration will be described in detail below according to other advantageous and / or more preferred embodiments of the method described herein.

[0015] In the preferred computer implementation method of this application, the heterogeneous source data (3) from multiple heterogeneous source applications (2) is integrated and analyzed, including the following steps: Execute the computational data acquisition program (1) to access and load heterogeneous source data (3) from at least one heterogeneous source application (2); Provide specifications for the heterogeneous data integration module (4) to convert heterogeneous source data (3) into target data (5); At least perform data analysis and processing based on the target data (6) to generate output data (7); Write the output data (7) into the normalized database (8).

[0016] The normalized database (8) is used to store normalized data and is configured to be accessible to users through any known user interface. Therefore, users can query the normalized database (8) to access structured and standardized data from different heterogeneous information sources, which would otherwise be non-standard and difficult to access and query effectively.

[0017] Furthermore, the output data (7) exists in a normalized form, and therefore can also be used to automatically generate reports based on the nature and purpose of the data analysis and processing (6). Such analysis reports are accessible to end users.

[0018] This approach enables the continuous integration of new heterogeneous data sources into a normalized database over time without requiring much additional work, as data can be collected from new “provider” / heterogeneous source applications (2) simply by retraining or reconfiguring the computational data acquisition program.

[0019] In one embodiment, the computational data acquisition procedure (1) is implemented through Robotic Process Automation (RPA) for building, deploying, and managing software robots that interact with the heterogeneous source application (2). Specifically, the software robot is responsible for converting information from the heterogeneous source application (2), such as web files or web pages, into a database model in a specific format that can be used more directly for subsequent extraction processes aimed at generating reports.

[0020] The heterogeneous data integration module (4) includes specifications adapted to the following operations: Input the first-level data structure specification, which describes the intermediate representation of the data, and the second-level data structure specification, which describes the representation of the target data (5); Implement the extraction framework specification to extract data from heterogeneous source data (3) to generate an intermediate representation of the data; Implement a data normalization framework specification to transform intermediate representations of data into target data (5).

[0021] In another embodiment of the method of this application, the extraction framework specification includes: parsing the first-level data structure specification to convert the extracted heterogeneous source data into an intermediate representation of the data. More specifically, the heterogeneous source data (3) is extracted and integrated using the extraction framework, which can be developed using the Python language and is designed to extract and store highly structured data so that it can become the object of more in-depth analysis using the data normalization framework specification.

[0022] In another embodiment of the method of this application, the data normalization framework specification includes executing a natural language processing computation program, which is configured to: Based on the first-level data structure of the intermediate representation of the data, a logical structure diagram is created, which is defined by multiple smaller data components. Execute the information extraction and calculation program to detect and classify key information data; Parse the second-level data structure specification to convert key information data into target data (5).

[0023] More specifically, to support automated data analysis and processing, a data normalization framework such as spaCy can be used, supported by natural language processing for handling natural language data. Therefore, in one example, when heterogeneous source data involves web pages containing text information, the natural language text is first processed, segmenting it into words, punctuation, etc., and then applications such as named entity recognition models / heuristics are implemented to identify "real-world" objects / entities (e.g., companies, people / athletes, objects, etc.).

[0024] One of the problems with the information quality provided by such heterogeneous source applications (2) lies in the non-uniqueness of the referenced entities: the same real entity may use different names, slightly different appellations, etc. To solve this problem, a search mechanism can be introduced to identify cases where there are different names in the process but they are actually the same entity, based on specific heuristic rules.

[0025] In another embodiment of the method of this application, the natural language processing computation program includes a natural language processing heuristic algorithm, which is constructed as follows: Perform text preprocessing on the intermediate representation of the data; Identify table information; Extract key information data.

[0026] In one embodiment of the method of this application, the text preprocessing heuristic algorithm may include: Separate numbers or letters from unrelated symbols or punctuation; Remove extra spaces or paragraphs; For each unit of measurement symbol, separate the unrelated characters before and after it.

[0027] In another embodiment of the method of this application, the Camelot extraction heuristic algorithm can be used to identify table information, for example: Validate the list; Check if the number of columns and rows in the table exceeds a preset threshold; Retrieve header keywords; and Extract the table entries.

[0028] Furthermore, in another embodiment of the method of this application, extracting key information data may include executing the following algorithm sequence: Fuzzy matching; Entity rule extraction; Entry building; and Entry verification.

[0029] In another embodiment of the method of this application, a computational data acquisition program (1) may be executed periodically.

[0030] In another embodiment of the method of this application, the computational data acquisition program (1) can be implemented by file transfer protocol (FTP) or web crawling.

[0031] In another embodiment of the method of this application, the data analysis and processing (6) is a pattern or feature recognition algorithm, and the method of this application further includes: Provide a user interface platform (9) having input components for generating user profile data (10); Data analysis and processing (6) are performed based on at least the target data (5) and user profile data (10) to generate output data (7).

[0032] More specifically, the user profile data (10) may include user preference data, and the data analysis processing (6) is configured to identify data items with features that match the user preference data in the target data, and the output data (7) is a set of feature data items that match the user preference data.

[0033] In another embodiment of the method of this application, the data analysis processing (6) involves a business intelligence algorithm.

[0034] In another embodiment of the method of this application, the method further includes: Access a translation service configured to dynamically translate output data (7) into a pre-configured language; optionally, the translation service may be a cloud service.

[0035] Specifically, data flows published on online heterogeneous source applications are typically written in the original language of the entity publishing the data. To facilitate understanding of such information, the method of this application translates the text from the original language into a user-configured language. This translation process is performed through a real-time cloud translation service, which offers higher content reliability compared to dictionary-based translation processing. As a result, all reports generated from the output data (7) produced by the method of this application are presented in a language easily understood by the end user, providing significant added value.

[0036] For those skilled in the art, it should be understood that the present invention is not limited to the above embodiments. Various modifications and variations can be made to it without departing from the spirit and essence of the present invention, and all such modifications and variations should fall within the protection scope of the present invention.

[0037] Of course, the preferred embodiments shown above can be combined in different possible forms, and all these combinations will not be described in detail here.

Claims

1. A computer-implemented method of integration and analysis of heterogeneous source data (3) for processing heterogeneous source data (3) acquired from a plurality of heterogeneous source applications (2), the method comprising the steps of: executing a computational data acquisition procedure (1) to access and load heterogeneous source data (3) from at least one heterogeneous source application (2); providing a heterogeneous data integration module (4) with specifications for converting the heterogeneous source data (3) into target data (5); implementing a data analysis process (6) based at least on the target data (5) to generate output data (7); writing the output data (7) to a normalized database (8); wherein providing the heterogeneous data integration module (4) with specifications comprises: inputting a first level data structure specification describing an intermediate representation of data and a second level data structure specification describing a representation of target data; implementing an extraction framework specification to parse the first level data structure specification to extract data from the heterogeneous source data (3) to generate the intermediate representation of data; implementing a data normalization framework specification to convert the intermediate representation of data into the target data (5).

2. The method according to any of the preceding claims, wherein, the data normalization framework specification comprises executing a natural language processing computational procedure configured to: create a logical structure graph based on the first level data structure describing the intermediate representation of data, the logical structure graph defined by a plurality of smaller data elements; execute an information extraction computational procedure to detect and perform classification on key information data; parse the second level data structure specification to convert the key information data into the target data (5).

3. The method of claim 2, wherein, the information extraction computational procedure is a named entity recognition procedure.

4. The method according to the preceding claims 2 and 3, wherein, the natural language processing computational procedure comprises implementing a natural language processing heuristic algorithm structured to: perform text pre-processing on the intermediate representation of data; identify tabular information; extract key information data.

5. The method of claim 4, wherein, the heuristic algorithm for performing text pre-processing on the intermediate representation of data comprises: separating numbers or letters from irrelevant symbols or punctuations; removing extraneous spaces or paragraphs; separating irrelevant characters before and after each unit of measurement symbol.

6. The method of claim 4 or 5, wherein, the heuristic algorithm for identifying tabular information is a Camelot heuristic algorithm structured to: verify a table; detect whether the number of columns and rows of the table exceeds a pre-set threshold; obtain table header keywords; and extract table entries.

7. The method of any one of claims 4 to 6, wherein, extracting key information data comprises implementing a sequence of algorithms comprising: fuzzy matching; entity rule extraction; entry construction; and entry verification.

8. The method according to any of the preceding claims, wherein, the computational data acquisition procedure (1) is executed periodically.

9. The method according to any of the preceding claims, wherein, the computational data acquisition procedure (1) is implemented through a file transfer protocol or web scraping process.

10. The method according to any of the preceding claims, wherein, the data analysis process (6) is a feature recognition algorithm.

11. The method of any one of the preceding claims, further comprising: providing a user interface platform (8) having input components for generating user profile data (9); The data analysis process is implemented based on at least the target data (5) and the user profile data (9) to generate output data (7).

12. The method of claims 10 and 11, wherein, The user profile data (8) comprises user preference data, and The data analysis process (6) is configured to identify in the target data features matching the user preference data, the output data (7) being a set of feature data matching the user preference data.

13. The method of any of the preceding claims, wherein, The data analysis process (6) involves a business intelligence algorithm.

14. The method according to any of the preceding claims, further comprising: accessing a translation service, the translation service being configured to dynamically translate the output data (7) into a preconfigured language, optionally, the translation service is a cloud service.

Images