Method for integrating and analysing heterogenous source data

Abstract

The present application relates to methods for integrating and analysing heterogenous data (3) from heterogenous source applications (2), in order to provide a solution to the denormalization inherent to the data models native to web-based database platforms, each one following a particular data configuration and, therefore, not respecting a single and standardized way of presenting data, not favouring a structured aggregation of information. To achieve this goal, the present application describes a computer- implemented method comprising: executing a computational data collection routine (1) to access and load heterogenous source data (3); providing an heterogenous data integration module (4) for transforming heterogeneous source data (3) into target data (5); implementing a data analysis process (6) based at least on target data (5), to generate output data (7); and feeding a normalized database (8) with output data (7).

Description

[0001] DESCRIPTION

[0002] METHOD FOR INTEGRATING AND ANALYSING HETEROGENOUS SOURCE DATA

[0003] FIELD OF THE APPLICATION

[0004] The present application is enclosed in the field of integration of data from heterogeneous source applications. More specifically, the present application relates to methods for integrating and analysing heterogenous data from heterogenous source applications.

[0005] PRIOR ART

[0006] Heterogeneous data are any data with high variability of data types and formats. Data from distinct source applications are often heterogenous because they stem from independent and disparate activities and are managed and maintained by different application owners. Therefore, these data often differ both in their values and in their structures, even if they relate to the same phenomena.

[0007] The ability to deal with data heterogeneity in an effective and efficient way is of utmost importance for data integration systems and a prerequisite to several other applications. For instance, when focusing on the Web of data, it enables semantic search in terms of entities and relations on top of the Web of text and deep reasoning using related ontologies, thus creating the Web of knowledge.

[0008] Therefore, when dealing with multiple heterogeneous data source applications, the final aim is often to fuse the different manifestations of the same real- world entity to get a unified view that gives users the illusion of interacting with one single data source. Currently, databases used for design and engineering employ a variety of different data models, interface languages, naming conventions, data semantics, schemas, and data representations. Thus, a fundamental problem for concurrent engineering is the sharing of heterogeneous information among a variety of design resources. Successful concurrent engineering also requires access to data from multiple stages of the design life-cycle, but the diversity among data from different tools and at different stages creates serious barriers.

[0009] The present solution intended to innovatively overcome such issues.

[0010] SUMMARY OF THE APPLICATION

[0011] It is therefore the object of the present application to be a solution to the denormalization inherent to the data models native to web-based database platforms, each one following a particular data configuration and, therefore, not respecting a single and standardized way of presenting data, not favouring a structured aggregation of information.

[0012] To achieve this goal, the present application describes a computer- implemented method for integrating and analysing heterogenous source data obtained from a plurality of heterogeneous source applications, which comprises the steps of:

[0013] - executing a computational data collection routine to access and load heterogenous source data from at least one heterogeneous source application;

[0014] - providing an heterogenous data integration module with specifications for transforming heterogeneous source data into target data;

[0015] - implementing a data analysis process based at least on target data, to generate output data;

[0016] - feeding a normalized database with output data. More specifically, the step of providing an heterogenous data integration module with specifications comprises:

[0017] - inputting a first level data structure specification describing an intermediate representation of data, and a second level data structure specification describing a target data representation;

[0018] - implementing an extraction framework specification to extract data from the heterogeneous source data in order to generate an intermediate representation of data;

[0019] - implementing a data normalization framework specification to transform the intermediate representation of data into target data.

[0020] The proposed solution uses and orchestrates the simultaneous operation of a set of innovative procedures based on, for example, Robotic Process Automation, Natural Processing Language and Artificial Intelligence mechanisms in order to carry out the collection of data from the various heterogeneous source applications, their processing and analysis, as well as a set of advanced data analysis aimed at providing aggregated information to a user.

[0021] The synchronized and automated operation of all these elements with the aim of producing knowledge encapsulates the innovative nature of the method described in the present application, as well as represents a technical challenge in the sense of putting all the elements in an effective and efficient dialogue with each other.

[0022] DESCRIPTION OF FIGURES

[0023] Figure 1 - a block diagram illustrating a first exemplary chain of data elements according to the method described in the present application. The reference sign represents:

[0024] 1 - computational data collection routine; 2 - heterogeneous source application;

[0025] 3 - heterogeneous source data;

[0026] 4 - heterogenous data integration module;

[0027] 5 - target data;

[0028] 6 - data analysis process;

[0029] 7 - output data;

[0030] 8 - normalized database.

[0031] Figure 2 - a block diagram illustrating a second exemplary chain of data elements according to the method described in the present application. The reference sign represents:

[0032] 1 - computational data collection routine;

[0033] 2 - heterogeneous source application;

[0034] 3 - heterogeneous source data;

[0035] 4 - heterogenous data integration module;

[0036] 5 - target data;

[0037] 6 - data analysis process;

[0038] 7 - output data;

[0039] 8 - normalized database;

[0040] 9 - user interface platform;

[0041] 10 - user profile data.

[0042] DETAILED DESCRIPTION

[0043] The more general and advantageous configuration of the method described in the present application are presented in the Summary of the application. Such configuration is detailed below in accordance with other advantageous and / or preferred embodiments of implementation of the method herein described. In a preferred aspect of the computer-implemented method described in the present application, the integration and analysis of heterogenous source data (3) obtained from a plurality of heterogeneous source applications (2), involves:

[0044] - executing a computational data collection routine (1) to access and load heterogenous source data (3) from at least one heterogeneous source application (2);

[0045] - providing an heterogenous data integration module (4) with specifications for transforming heterogeneous source data (3) into target data (5);

[0046] - implementing a data analysis process (6) based at least on target data, to generate output data (7);

[0047] - feeding a normalized database (8) with output data (7).

[0048] The normalized database (8) is used to store normalized data, and it is configured to be accessible to a user, through any known user-interface means. Consequently, a user is able to consult the database (8) to access structured and standardized data, coming from different heterogenous sources of information, which, otherwise, would be non-standardized, making accessing and consulting information more effective.

[0049] Additionally, the output data (7) being in a normalized fashion, may also be used to prepare automatic reports depending on the nature and purpose of the data analysis process (6). Such analysis reports may then be accessible to an end-user.

[0050] The developed method makes it possible to integrate new sources of heterogeneous data into a normalized database over time and without considerable additional effort, since it is only necessary to retrain the computational data collection routine to collect data from the new "Provider" / heterogeneous source application (2).

[0051] In one embodiment, the computational data collection routine (1) is implemented using a robot process automation to build, deploy and manage software robots to interact with heterogeneous source applications (2). More particularly, the robots are responsible for converting the information from heterogeneous source applications (2), such as web files or web pages, into a database model, in a specific format, that may be oriented towards a later extraction more directed towards the production of reports.

[0052] The heterogenous data integration module (4) includes specifications adapted to:

[0053] - input a first level data structure specification describing an intermediate representation of data, and a second level data structure specification describing a target data (5) representation;

[0054] - implementing an extraction framework specification to extract data from the heterogeneous source data (3) in order to generate an intermediate representation of data; and

[0055] - implementing a data normalization framework specification to transform the intermediate representation of data into target data (5).

[0056] In another embodiment of the method, the extraction framework specification comprises parsing the first level data structure specification to transform extracted heterogeneous source data into an intermediate representation of data. More specifically, the heterogeneous source data (3) is subject to an extraction and integration process using the extraction framework which may be developed in the python language, with the aim of extracting and storing data with a higher level of structure so that they can be object of deeper analysis, using data normalization framework specification.

[0057] In another embodiment of the method, the data normalization framework specification comprises executing Natural Language Processing computational routines configured to: - create a logical structure diagram from the first level data structure describing intermediate representation of data, defined by a plurality of smaller data components;

[0058] - execute an information extraction computational routine to detect and categorize essential information data;

[0059] - parse the second level data structure specification to transform essential information data into target data (5).

[0060] More specifically, to support an automatic data analysis process, a data normalization framework such as the spaCy may be used, supported by a Natural Language Processing process, which processes data in natural language. In this sense, in one example, where the heterogeneous source data relates to a web page comprised by textual information, the text in natural language is first processed in order to be segmented into words, punctuation, etc., followed by the implementation of the application of, for example, Named Entity Recognition models / heuristics aimed at identifying "real world" objects / entities (e.g. companies, people / players, objects, etc.).

[0061] One of the existing problems in the quality of information that is made available on such heterogeneous source applications (2) is the question of the nonuniqueness of the entities that are referenced; The same real entity can be referred to with different nomenclatures, subtle differences in designation, etc. In order to circumvent this issue, a searching mechanism may be included being to find, based on certain heuristics, the cases in which entities referenced in the procedures are actually the same.

[0062] In another embodiment of the method, the Natural Language Processing computational routines include implementing Natural Language Processing heuristics adapted to: text pre-processing intermediate representation of data; identify table information; extract essential information data.

[0063] In one embodiment of the method, text pre-processing heuristics may include:

[0064] - separating numbers or letter from unintended symbols or punctuation;

[0065] - removing multiple spaces or paragraphs;

[0066] - separating previous / following unintended character for each unit measure symbol.

[0067] In another embodiment of the method, a Camelot extraction heuristic may be used to identify table information, such as:

[0068] - verify column tables;

[0069] - verify if the tables have more columns and rows than a predefined threshold;

[0070] - obtain header keywords; and

[0071] - extract table items.

[0072] Additionally, in another embodiment of the method, extracting essential information data may include implementing the following sequence of algorithms:

[0073] - Fuzzy matching;

[0074] - Entity ruler extraction;

[0075] - Item construction; and

[0076] - Item verification.

[0077] In another embodiment of method, the computational data collection routine (1) is executed periodically.

[0078] In another embodiment of method, the computational data collection routine (1) is implemented via a File Transfer Protocol or a web Scraping process. In another embodiment of method, the data analysis process (6) is a pattern recognition algorithm and the method further comprises:

[0079] - providing a user interface platform (9) with input means for generating user profile data (10);

[0080] - implementing the data analysis process based at least on target data (5) and on the user profile data (10), to generate output data (7).

[0081] More specifically, user profile data (9) may include user-preference data, and wherein, the data analysis process (6) is configured to identify patterns in target data matching with user-preference data; the output data (7) being data patterns that match the user-preference data.

[0082] In another embodiment of method, the data analysis process (6) relates to Business Intelligence algorithms.

[0083] Finally, in another embodiment of method, it further comprises:

[0084] - accessing to a translation service adapted to dynamically translate output data (7) into a preconfigured language, optionally, the translation service is a cloud service.

[0085] Specifically, data procedures published on online heterogeneous source applications are usually written in an original language of the entities that publish them. In order to facilitate the understanding of said information, the method carries out a translation of the texts from the original language into the configured language of a user. This translation process is carried out using real-time translation Cloud services, which allow for superior content reliability when compared to translation processes based on word dictionaries. In this way, all reports that may be produced based on output data (7) produced by the method are presented in a language that is easy for the end user to understand, which is an obvious added value. As will be clear to one skilled in the art, the present invention should not be limited to the embodiments described herein, and a number of changes are possible which remain within the scope of the present invention. Of course, the preferred embodiments shown above are combinable, in the different possible forms, being herein avoided the repetition all such combinations.

Claims

CLAIMS1. Computer-implemented method for integrating and analysing heterogenous source data (3) obtained from a plurality of heterogeneous source applications (2), comprising the steps of:- executing a computational data collection routine (1) to access and load heterogenous source data (3) from at least one heterogeneous source application (2);- providing an heterogenous data integration module (4) with specifications for transforming heterogeneous source data (3) into target data (5);- implementing a data analysis process (6) based at least on target data (5), to generate output data (7);- feeding a normalized database (8) with output data (7); wherein, providing an heterogenous data integration module (4) with specifications comprises:- inputting a first level data structure specification describing an intermediate representation of data, and a second level data structure specification describing a target data representation;- implementing an extraction framework specification to parse the first level data structure specification to extract data from the heterogeneous source data (3) in order to generate an intermediate representation of data;- implementing a data normalization framework specification to transform the intermediate representation of data into target data (5).

2. The method according to any of the previous claims, wherein the data normalization framework specification comprises executing Natural Language Processing computational routines configured to:- create a logical structure diagram from the first level data structure describing intermediate representation of data, defined by a plurality of smaller data components;- execute an information extraction computational routine to detect and categorize essential information data;- parse the second level data structure specification to transform essential information data into target data (5).

3. The method according to claim 2, wherein the information extraction computational routine is a Named Entity Recognition routine.

4. The method according to the previous claims 2 and 3, wherein the Natural Language Processing computational routines include implementing Natural Language Processing heuristics adapted to:- text pre-processing intermediate representation of data;- identify table information;- extract essential information data.

5. The method according to claim 4, wherein the heuristic to text pre- process intermediate representation of data includes:- separating numbers or letter from unintended symbols or punctuation;- removing multiple spaces or paragraphs;- separating previous / following unintended character for each unit measure symbol.

6. The method according to claim 4 or 5, wherein the heurist to identify table information is a Camelot heuristic adapted to:- verify column tables;- verify if the tables have more columns and rows than a predefined threshold;- obtain header keywords; and- extract table items.

7. The method according to any of the claims 4 to 6, wherein extracting essential information data includes implementing the following sequence of algorithms:- Fuzzy matching;- Entity ruler extraction;- Item construction; and- Item verification.

8. The method according to any of the previous claims, wherein the computational data collection routine (1) is executed periodically.

9. The method according to any of the previous claims, wherein the computational data collection routine (1) is implemented via a File Transfer Protocol or a web Scraping process.

10. The method according to any of the previous claims, wherein the data analysis process (6) is a pattern recognition algorithm.

11. The method according to any of the previous claims, further comprising:- providing a user interface platform (8) with input means for generating user profile data (9);- implementing the data analysis process based at least on target data (5) and on the user profile data (9), to generate output data (7).

12. The method according to claims 10 and 11, wherein user profile data (8) includes user-preference data, and wherein, the data analysis process (6) is configured to identify patterns in target data matching with user-preference data; the output data being data patterns that match the user-preference data.

13. The method according to any of the previous claims, wherein the data analysis process (6) relates to Business Intelligence algorithms.

14. The method according to any of the previous claims, further comprising:- accessing to a translation service adapted to dynamically translate output data into a preconfigured language, optionally, the translation service is a cloud service.

Images