Data enrichment system

Abstract

A computer-implemented method for enriching data according to a characteristic is provided. The method uses data content from at least two data sources, each comprising at least one data type, and includes assigning a confidence factor to one data source for the characteristic (221), preprocessing the data received from the data source according to a data type (222), analyzing the information contained in the data concerning the characteristic (223), and applying an enrichment process to select the information for data enrichment related to the characteristic based on the information analysis and the confidence factor (224). Figure for abstract: Fig. 2.

Description

Title of the invention: Data enrichment system

[0001] The present disclosure relates generally to retrieving and processing data in a distributed computing system, in particular enriching data according to a characteristic using data content from at least two data sources comprising at least one data type. Background

[0002] Data retrieval and processing in a distributed environment faces many problems. When multiple data sources need to be searched to retrieve information, many problems arise. Different data sources may store different types of data, provide different information content, and / or require different search parameters, if they are searchable at all. In addition, the reliability of data sources may also differ in general or even particularly regarding the specific information required. Therefore, data retrieval for end users may be time-consuming since they have to find the data sources providing the required content, search with different parameters or only visually search the data sources, install different programs to do searches and so on.In addition, end users may end up with incorrect information due to the lack of reliability of the data source or the content shared by the data source. Finally, an end user who has to search multiple data sources (and cannot retrieve relevant information from some of them) wastes a lot of computing resources and bandwidth.

[0003] For example, a traveler wants to search for experiences and services in a destination (e.g., tours, activities, attractions), and current digital providers of customer-facing information are not travel-specific (e.g., Google Maps), not comprehensive (e.g., only related to travel to / from the destination), or sometimes not accurate (e.g., platforms that rely on user reviews such as TripAdvisor™). Destination content is extremely fragmented and sometimes related information is outdated since it depends on the content owner's willingness and ability to update the various sources / platforms. This makes it difficult for travelers to search for appropriate activities.

[0004] Therefore, from the perspective of time and resource consumption as well as accuracy, it is necessary to have a data source that provides all relevant information to an end user. This goal can be achieved by a data enrichment process that refers to the process of adding additional information or context to a data set in order to improve its quality. However, a data enrichment process in a distributed environment faces the same problems as an end user searching for relevant information.

[0005] Therefore, it is necessary to improve the data enrichment process in a distributed environment. Summary

[0006] According to a first aspect, a computer-implemented method of enriching data according to a characteristic is provided. The method uses data content from at least two data sources comprising at least one data type and comprises assigning a confidence factor to a data source for the characteristic, preprocessing data received from the data source according to a data type, analyzing information included in the data regarding the characteristic, applying an enrichment process to select information for data enrichment related to the characteristic based on the analysis of the information and the confidence factor

[0007] In some embodiments, at least one data type includes one or more of text, structured information, image, audio, and video. In other embodiments, the confidence factor reflects the reliability of the data source with respect to the characteristic. In another embodiment, the confidence factor is determined based on a machine learning model that is continuously trained with data retrieved from the two or more data sources.

[0008] In some embodiments, preprocessing the data comprises extracting the content of the data by at least one of, a) for the text data type: applying natural language processing for translation, correction, formatting, speech tagging and / or extracting features from the text, b) for the structured information data type: organizing the content according to the tags, c) for the image data type: applying at least one of object detection, face detection, object segmentation and object recognition, d) for the sound data type: applying speech recognition and / or natural language processing, and e) for the video data type: applying at least one of object detection, face detection, speech recognition and natural language processing.

[0009] In other embodiments, the data preprocessing includes extracting metadata from the data. In other embodiments, the metadata includes at least one of the descriptive metadata concerning at least one of the title, subject, genre, and author; the rights metadata concerning at least one of the title, copyright status, rights holder, and license conditions; the technical metadata concerning at least one of the file types, size, creation date, creation time, compression type, uniform resource locator, and page ranking; the preservation metadata concerning a location of an article in a hierarchy or in a sequence; and the image metadata concerning at least one of a timestamp, camera properties, resolution, size, and geotag.

[0010] In some embodiments, analyzing the information comprises at least one of a) for the text data type: applying natural language processing to analyze the similarity of the authors' words and / or sentiments, b) for the image data type: analyzing detected objects and / or faces regarding at least one of facial expression, emotion, age, and gender, and c) for the structured data type: discovering rules in the structured data.

[0011] In other embodiments, analyzing the information comprises determining a relevance score for the data with respect to the feature and a confidence value in the accuracy of the relevance score.

[0012] In some embodiments, the enrichment process is based on a machine learning model, wherein the input to the machine learning model includes the relevance score for the data with respect to the feature, the confidence value in the accuracy of the information, the confidence factor of the data source of the information, a weighting factor of the information, and a timestamp. In other embodiments, the input to the machine learning model includes other information extracted from the metadata. In other embodiments, the enrichment process for selecting the data enrichment information includes determining a probability value of accuracy of the information for enriching the data and, in response to the probability value being higher than a threshold value, selecting the information.

[0013] In some embodiments, the method is triggered periodically and / or in response to a request for recommendations regarding the characteristic.

[0014] According to a second aspect, a data enrichment system configured to perform the method, as described herein, is presented.

[0015] According to a third aspect, a computer program product comprising program code instructions stored on at least one computer-readable medium for performing the method as described herein is presented, when said program code instructions are executed on a computer. Brief Description of the Drawings

[0016] The foregoing and hereinafter objects, features and advantages of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings, in which like numerals are used to represent like elements, wherein:

[0017] [Fig.l] shows an overview of a computer-implemented data enrichment method.

[0018] [Fig.2] shows a flowchart of an example or data enrichment.

[0019] [Fig.3] describes examples of data sources and contents.

[0020] [Fig.4A] shows data source confidence factors on a data type and feature level; [Fig.4B] shows data source confidence factors on a feature level.

[0021] [Fig. 5A] shows an example of photo content preprocessing; [Fig. 5B] shows an example of text content processing.

[0022] [Fig. 6A] shows an example of photo content analysis after preprocessing; [Fig. 6B] shows an example of text content analysis after preprocessing.

[0023] [Fig. 7] describes an enrichment process.

[0024] [Fig. 8] is another detailed example of data enrichment according to the disclosure.

[0025] [Fig. 9] shows a schematic representation of a computer system implementing the functionalities described herein. Detailed Description

[0026] The present disclosure relates to methods and systems for enriching data according to a characteristic using the data content of at least two data sources comprising at least one type of data. Generally, data enrichment processes aim to enrich the data of a data source, which may be stored in distributed databases or in a single database. In other words, data enrichment refers to the process of adding additional information or context to a set of data in order to improve its quality. Data enrichment can be an easy task when the information to be added is contained in a reliable data source that includes only one type of data.

[0027] However, in the current world where this information is distributed across the entire Internet in different data formats provided by different data sources, a data enrichment process faces many issues: How reliable is a data source and / or how reliable is that data source with respect to a specific characteristic to be researched? What types of data are provided and how can the relevant information for data enrichment be retrieved? The answers to these questions and the issues arising therefrom are addressed by the data enrichment process described herein.

[0028] It should be noted that, the present application describes the data enrichment process in a travel-related context, the data enrichment process as described herein can be applied wherever enrichment of data from different data sources is required.

[0029] In the travel industry, the applicability of the present disclosure is very visible. The pandemic has reinforced two fundamental elements of the travel industry. First, travel has become deeply personalized. This means that it is necessary to offer unique experiences to customers with different preferences. Second, each experience is considered by the traveler as a complete trip, and not a series of isolated events.

[0030] As a traveler, it is not always easy to find the information that will facilitate decision-making. A traveler may need to find out if the activity is suitable for specific categories, such as age (e.g. children, teenagers, seniors), reduced mobility, animals, if the activity is easily accessible by public transport, proximity to parking, regarding languages ​​spoken, if the activity has specific characteristics, such as seasonality (e.g. outdoors only in summer), weather dependence, equipment (needed to do the activity), triggered emotions, price range, conditions, recommended average visit time, popularity by country / culture, or if it is a good place to take pictures, if the activity is similar to other activities, if the provider's trust level is high, or which photos are most representative of the activity.

[0031] Now, 90% of travelers seek personalized travel recommendations when planning their trips, as time and resources are always an issue. Consequently, travelers demand highly personalized content based on their profiles and interests. This hyper-personalization means that travel providers understand their customers as micro-segments, rather than as monoliths, i.e., going beyond categories such as business / leisure, family / solo travelers, etc. The computer systems on which travel providers rely have detailed travel-related information on the content that can be recommended to the micro-segments.

[0032] When it comes to recommending destination experiences and services (tours, activities, attractions) to travelers, current digital providers of customer information are not travel-specific (e.g., Google Maps), not comprehensive (e.g., only related to travel from / to the destination), or sometimes not accurate (e.g., those based on user reviews and recommendations such as TripAdvisor™ or Google Maps). Destination content is extremely fragmented and sometimes the related information is outdated since it depends on the content owner's willingness and ability to update the various sources / platforms. This makes it difficult to leverage them in a travel context or to provide information to travelers or recommend activities, respectively.

[0033] Embodiments of the present disclosure describe methods for enriching data regarding destination content, such as stores, museums, activities. The process leverages different sources and applies information extraction techniques to provide travelers (e.g., via travel distributors) with updated information, further details that are relevant to the travelers, and / or information that is not accessible / visible to the user. The processes of the disclosure may be used to populate a catalog (list of activities) or enhance a recommendation engine.

[0034] This disclosure facilitates the creation of an enriched database of travel providers, which could be used as input to a recommendation system or searched by travelers. However, any other enriched database may benefit from the disclosure since it is also described for any enrichment method in general. Enrichment categories in the travel industry may particularly concern suitability (children, groups, pets, popularity, triggered emotion...), accessibility (parking, transportation, disability,...), seasonality and weather dependency, level of trust and timeliness, etc.

[0035] The benefits of disclosure in a travel-related context are ubiquitous. One goal is the creation of an enriched and updated catalog of destination experiences. This involves extracting new information from various raw data sources, creating new, elaborated information based on the extracted information, and adjudicating in the event of conflicting information based on the reliability of the data source (e.g., the supplier's official information states "not suitable for dogs" but customer photos on TripAdvisor show pictures of dogs). With the help of the methods and systems described herein, it is possible to create a highly effective recommendation system that meets the growing needs for hyper-personalization.

[0036] [Fig.l] presents an overview of a computer-implemented data enrichment method and the respective components. The main components presented herein are one or more input data sources 110, from which data are retrieved, a computer system 120, implementing the data enrichment method described herein, an output storage 130 which is enriched with information.

[0037] The data sources 110 include one or more data types, said one or more text types 111, structured information 112, image 113, audio 114, and video 115. For example, one kind of data 110 may provide text content 111 in the form of natural language text, such as user reviews, forum discussion, or blog posts. Another data source 110 may provide structured information content 112 in the form of a standardized structure, an organizational structure, in HTML structure, or the like.

[0038] A single data source 110 may also provide two or more different types of data, for example, text content 110 and image or photo content 113. The images 113 may relate to images of a user taken at a location or object, images accompanying a review, painted images, or the like. The sound content 114 may relate to music or other sound recordings, if no visual content is available. The video content 115 may also include sound but may also be silent. The videos 115 may be provided for events and attractions, for facility support, and the like.

[0039] The computer system 120 includes modules and components for different tasks. The computer system 120 comprises in particular a trust module 121, one or more pre-processing modules 122, one or more analysis modules 123 and an enrichment module 124.

[0040] The trust module 121 is configured to assign one or more trust factors to one or more data sources 110 for a specific characteristic, respectively. One characteristic relates to a characteristic of an event, an attraction, an object, an item of a person, or the like. The characteristic may be evaluated as being relevant for data enrichment since it is important to know it for end users. For example, if the data registration process is related to attractions, one characteristic that may be relevant for data enrichment is whether the attraction is suitable for dogs or not. Another characteristic may be whether the attraction is suitable for disabled or elderly people. Another characteristic may be whether the attraction can also be visited in stormy or rainy weather.

[0041] A confidence factor may be a factor between 0 and 1 (or 0 and 100%) but may also be defined in any other range. A confidence factor of 0 may represent a data source 110 that is not reliable when it comes to information related to the characteristic "suitable for dogs". A confidence factor of 1 may indicate that the information in that data source 110 is absolutely reliable with respect to the selected characteristic. In addition, the confidence factor may also depend on the data type. For example, a data source 110 providing different data types may be assigned a different confidence factor for each of the data types, e.g.a data source 110 providing text 111 and images 113 may have a confidence factor of 0.9 for the text for a feature and a confidence factor of 0.8 for the images for the same feature reflecting that the text content 111 from that data source is more reliable than the image content 113. .

[0042] The confidence factor may be determined, in some embodiments using a machine learning model. The machine learning model may be continuously trained with data retrieved from the data source(s) 110. Additionally or alternatively, the confidence factor may be determined by a feedback loop mechanism. Such a mechanism may involve a human supervisor validating the accuracy of the data or a data source, e.g., after a machine learning model or other model determines a confidence factor, via a user interface. The confidence factor or accuracy assigned by the human supervisor may also be used as additional data sources to feed and train the machine learning algorithm.

[0043] The preprocessing module 122 is configured to preprocess the data received from the data source according to a data type. The data received from the data source is processed in terms of observations, i.e., a single group of data, e.g., regarding text content 111, such as a user review, or image content 113, e.g., a taken photo, etc.

[0044] Preprocessing generally refers to extracting computer-readable data for further analysis. For example, for text content 111, preprocessing may include applying natural language processing for translation, correction, formatting, speech tagging, and / or extracting features from the text. For structured information 112, preprocessing may include organizing content according to tags. For 113 images, preprocessing may include object detection, face detection, object segmentation, and / or object recognition. For the audio data type 114, the preprocessing may include the application of speech recognition and / or natural language processing. For videos 115, the preprocessing may include the application of object detection, face detection, speech recognition, and / or natural language processing.

[0045] The preprocessing may also, additionally or alternatively, include extracting metadata from the data retrieved from data sources 110. The metadata may include at least one of descriptive metadata regarding at least one of title, subject, genre, and author, rights metadata regarding at least one of titles, copyright status, rights holder, and license terms, technical metadata regarding at least one of file types, size, creation date, creation time, compression type, uniform resource locator, and page ordering, preservation metadata regarding an article location in a hierarchy or sequence, and image metadata regarding at least one of a timestamp, camera properties, resolution, size, and geotag.Metadata may, for example, indicate whether a photo 113 or video 115 was recently taken and may be a current impression and may have been taken in a specific location. Text metadata 110 may indicate whether and by whom the text was written. .

[0046] The preprocessing may be performed by applying multiple preprocessing modules 121, wherein a preprocessing module 122 applies a model that is specific to a data type. There may also be preprocessing modules 122 that preprocess multiple data types. One or more preprocessing modules 122 may be based on one or more machine learning models that have been trained according to the data type and the required data output. For example, one machine learning model may be trained to classify humans and / or animals in the images 113. Another that may be used by the same preprocessing module 121 may be trained to find faces of humans or any predefined objects.

[0047] The preprocessed data is then processed by the analysis module 123. The analysis module 123 is configured to analyze information included in the preprocessed data regarding the feature. For example, if the feature is "suitable for dogs," the analysis module 123 may determine for an observation, e.g., a user review, whether it includes information about dogs.

[0048] The analysis of the information may vary depending on the type of data. For example, for the text 111, the analysis of the information may include applying natural language processing to analyze word similarity and / or sentiment of the authors. For the image data type 113, the analysis of the information may comprise the analysis of detected objects and / or faces regarding at least one of facial expression, emotion, age and gender. For the structured data 112, the analysis of the information may comprise type discovery rules in the structure data. Of course, the application of natural language processing may also be applied for the audio 114 and the detection of faces regarding at least one of facial expression, emotion, age and gender may also be applied for the videos 115. In principle, the same analysis processes may be applied to different types of data, if appropriate and known to the skilled person.

[0049] There may be several analysis modules 123 each dedicated to one or more types of data. One or more of the analysis modules 123 may be based on machine learning models. Additionally or alternatively, analyzing the information may include determining a relevance score for the data with respect to the feature and a confidence value in the accuracy of the relevance score. The relevance score may be a value from the set {-1,0,l], where -l means does not meet the feature, 1 means meets the feature, and 0 means no specific information regarding the feature. Alternatively, the relevance score may also be any value in an interval [-1; 1] with a similar meaning as mentioned above. Other sets or intervals may be implemented as well.The confidence value may indicate how certain the analysis module 123 is of the relevance score. .

[0050] For example, the analysis module 123 may implement a machine learning model to determine whether a photo, which preprocessing has revealed to be of a dog, shows a relevant scene of the activity, e.g., the dog in a museum or the like. If this is found to be relevant to the feature and a dog is shown, the relevance score would be 1. The machine learning model may also indicate that this assessment is very confident, e.g., 89%.

[0051] Finally, the enrichment module 124 receives at least the outputs of the analysis modules 123 and the confidence module 121. In addition, the enrichment module 124 may also receive at least a portion of the output of the preprocessing module 122, as shown in [Fig.l] with a dotted line. The enrichment module 124 applies an enrichment process to select information for data enrichment, related to the characteristic based on the analysis of the information and confidence factor.

[0052] For a given characteristic, the information may be evaluated in the enrichment module 124 by taking into account each atomic source of information that may be provided by the other module included in the computer system 120, such as the predicted relevance score, the confidence value, the confidence factor and metadata, such as timestamps (creation or update of the information indicating freshness) or other extracted information included in the metadata.

[0053] In one embodiment, the enrichment process 124 may implement a machine learning model, wherein the input to the machine learning model includes the relevance score for the data relative to the feature, the confidence value in the accuracy of the information, the confidence factor of the data source of the information, a weighting factor of the information, and a timestamp. The information is provided for each observation that is processed. The weighting factor may be a weight assigned to the observation. For example, the weighting factor may be higher for one type of data, e.g., text content 111, than another type of data, e.g., audio 114.

[0054] The enrichment process may further or alternatively include determining a probability value for the enrichment data information to be correct. For example, a machine learning model may be trained to evaluate, based on information about a set of observations from different data sources and with different data types, whether one meets the characteristic or not, and then determine a probability value that reflects the machine learning model's level of certainty about whether one meets the characteristic or not. For example, if the characteristic is "suitable for dogs," the machine learning model may determine that most data sources and observations indicate that this is "TRUE" for this activity, the output of the enrichment process may be a 79% probability that this "TRUE"-information is correct.

[0055] In response to a probability value higher than a threshold value, the enrichment process may include selecting information for enriching the data. In the above example, the threshold value may be considered to be 75%, i.e., only if the machine learning model is more than 75% certain of its decision, the data 131 belonging to the activity in the output storage 130 will be enriched. This means that, if the machine learning model determines a value of 79%, the data belonging to the respective activity (e.g., the name, address, conditions, etc. provided by the owner of the activity) can be enriched with the information "suitable for dogs: yes".

[0056] Although [Fig. 1] shows many components and modules, not all of them are required to implement the basic method, which is shown in [Fig. 2] as is known to those skilled in the art. [Fig. 2] begins with assigning a confidence factor to a data source for a characteristic in box 221. The confidence factor can be assigned in a variety of ways, by humans or models, such as machine learning or traditional models. The confidence factor reflects the trustworthiness of a data source 110 with respect to the characteristic. The method continues with box 222 which relates to preprocessing data received from a data source 110 according to a data type. Receiving should be understood broadly and may include sending or retrieving data, requesting data, or accessing the data source.

[0057] Next, the method analyzes information included in the data regarding the characteristic in box 223. The analysis may be based solely on the preprocessed data but may also use the original data as input. Finally, the method applies, in box 224, an enrichment process based on the analysis and the confidence factor. In some embodiments, other data may be considered as well, e.g., the data output of the preprocessing of box 222 or the original data. Although boxes 221, 222, 223, and 224 are presented in sequential order, the process of assigning a confidence factor to a data source may also be processed in parallel or independently of the execution of boxes 222 and 223.

[0058] [Fig. 3] depicts examples of data sources 300, such as data sources 110, that store different types of data 303. The data is organized into observations 301 that are groups of data belonging to an entry, e.g., a user review, a photo, or the like. For each data source 300, an observation 301 is highlighted in example 302. The selected feature is "dog-friendly" and the activity is Kayak ABC, e.g., a kayak adventure.

[0059] Website A provides three types of data, text reviews 310, photos 320 and the list of information 330 about the activity. For the text content, there are four observations 311, 312, 313, 314 available, for the photo content, there are three observations 321, 322, 323 available, and for the structured content, there is one observation 331 available. Website B also provides three types of data, text reviews 340, photos 350 and the list of information 360 about the activity. For the text content, there are two observations 341, 342 available, for the photo content, there are three observations 351, 352, 353 available, and for the structured content, there is another observation 361 available. Website C also provides one type of information, namely, owner photos 370 for which two observations 371,372 are available. Finally, a travel provider system may provide structured content 380 with two observations 381,382.

[0060] The example of text reviews from users of website A 310 displays a text review in French, while the example of text reviews from users of website B 340 displays a current review in English. Therefore, the language may vary. The text is also often misspelled, its grammar is exotic, and it may include imaginary words. For the algorithm to work, said text must be transformed into a common and clean standardized language.

[0061] The example of the photos of the users of Website A 320 shows a dog in a kayak. This is a good example that the dog would probably be allowed to participate in the adventure. In contrast, the example of the photos of the users of Website B 350 shows only the dog on the shore and the photos of the owner of Website C do not show any dogs. The method and system presented herein are capable of extracting this information to make a decision as to whether the activity is suitable for dogs or not.

[0062] The examples of the lists of Website A and Website B 330, 360 as well as the contents of the databases of the travel provider system 380 show that the structured data can also vary. It is not simple to extract the relevant information from the structured data itself.

[0063] Figs. 4A and 4B show the lists of confidence factors. [Fig.4A] shows confidence factors of data sources on a data type and a feature level. In Table 410, it can be seen that Website A and Website B provide kinds of data types each. The confidence factors for a feature vary for the data types. For example, the confidence factor of the text content of Website A for the feature "suitable for pets" is 0.63, while the confidence factor of the image content of Website A for the same feature is 0.82. Assigning a confidence factor individually for each data source and individually for the feature and for the data type allows for a more granular assignment of the confidence factors.Furthermore, the reliability of a data type can vary fundamentally within a data source, so such a confidence factor as shown in Table 410 can be advantageous when it comes to reliability of data enrichment later. .

[0064] [Fig.4B] shows confidence factors of data sources on a feature level. Here, it can be seen in Table 420 that two confidence factors were assigned to Website A, namely 0.7 for the feature “suitable for pets” and 0.8 for the feature “suitable for disabled people”. Website B, Website C and the travel provider system also have all different confidence factors for each feature. However, unlike Table 410 in [Fig.4A], there is no differentiation between data types. This reduces the computational effort for assigning confidence factors compared to the confidence factors in Table 410. In addition, the confidence factor in Table 420 is encoded with a single decimal place while the confidence factor in Table 410 has two decimal places, further reducing the memory required for storage.

[0065] A confidence factor per feature may be calculated for a data source once and may be periodically refreshed using the machine learning models trained based on information provided by the data source. In some examples, the model training may also incorporate a "ground truth" that reflects an actual accuracy value of a data source taken by another model or a human supervisor.

[0066] Figs. 5A and 5B show examples of preprocessing raw data content from data sources, such as data sources 110. [Fig. 5A] shows an example of preprocessing photo content, such as photo content 113. A photo 510 of a man and a dog in a kayak is shown. Photo 510 may be processed by one or more machine learning models for object detection / segmentation / recognition or the like. Object detection in images uses computer vision and may assign a probability, as explained in the following.

[0067] The machine learning model may detect a human being in a bounded box 511 and categorize the human being as an adult male, as displayed in box 512. The same or another machine learning algorithm may also detect an animal with bounded box 513 and may categorize the animal as a dog, as displayed in box 514. The same or another machine learning algorithm may also detect the human's face in bounded box 515 and may categorize it as a human's face, as displayed in box 516. The face may also be cropped and resized during preprocessing to facilitate further processing. Machine learning models can also output probability scores based on the probability of the object, e.g., number of objects, nature, segmentation size, etc.) and similarity scores. .

[0068] Additionally, the preprocessing may also include extracting metadata 517 from the photo. The photo metadata may include at least one of a timestamp, camera properties, resolution, size, and a geotag. In the example of [Fig.5A], the metadata includes (from top to bottom) a unique photo identifier, a timestamp, a photo author, a photo geotag, the photo size, and other information indicated by three dots.

[0069] [Fig.5B] shows an example of processing text content, such as text content 111. The user's review of 520 is written in German and includes spelling mistakes and imaginary words. The preprocessing may first correct the spelling mistakes and imaginary words as shown in bold in box 521. Then, the text may be translated into French for further processing as shown in box 522. Of course, the text may also be translated first and then corrected and both stages of preprocessing may be implemented by one or more machine learning algorithms. In addition to this cleaning and translation, certain features of the text may be calculated, such as entity name recognition, speech tagging and so on.

[0070] Speech tagging matches the word and part of speech based on definition and context, e.g., "she (PRON) eats (VERB) chicken (NOUN)". Text can also be filtered so that only nouns are kept, e.g., "the bakery makes bread" => "bakery, bread". Additionally or alternatively, text similarity to keywords can be calculated, e.g., "similarity(boat, kayak)=top".

[0071] Additionally, the preprocessing may also include extracting metadata 527 from the user review. The text metadata may include at least one of title, subject, genre, and author, rights metadata regarding at least one of title, copyright status, rights holder, and license terms. In the example of [Fig.5B], the metadata includes (from top to bottom) a unique identifier of the user review, a timestamp, a text author, and other information indicated by three dots.

[0072] Metadata for all data types may also include at least one of technical metadata related to at least one of file type, size, creation date, creation time, compression type, uniform resource locator, and page ordering and preservation metadata regarding an item's location in a hierarchy or sequence. The metadata may be defined individually for each preprocessing module and may depend on the data type and the data required for each characteristic.

[0073] Other data types may require different preprocessing. For example, in the travel industry, a PNR contains reservation information. The reservation information may include travel activities and local activities. The structured data of a PNR may be preprocessed so that the data relevant to data enrichment and other data are extracted, e.g., travel data per se, are omitted. For example, animal-related data, such as "animal in cabin" may be extracted but other data, such as flight number or the like, may be omitted. In another example, the video content may be preprocessed so as to extract single frames, the frames then being preprocessed as previously described with respect to [Fig.5A]. The sound data may be preprocessed by applying speech recognition and speech-to-text algorithms. Therefore, multiple variations of preprocessing may be applied in a data enrichment system that is capable of processing data from a plurality of data sources providing a plurality of data types.

[0074] Figs. 6A and 6B show examples of analyzing information of different data types according to embodiments of the disclosure. [Fig.6A] shows an example of photo content analysis after preprocessing. For each preprocessed observation as shown in box 610 with the preprocessed image 611 and metadata 612 a machine learning model 613 may be applied. The input may be, e.g., at least one classification(s) of objects found in the image, adjustments of the image, at least part of the extracted metadata, and so on. The input is processed, e.g., by analyzing detected objects and / or faces regarding at least one of facial expression, emotion, age, and gender.

[0075] The machine learning model 613 may be a supervised or semi-supervised model. The machine learning model 613 may be one of a gradient boosted tree, a random forest, an artificial neural network, a recurrent neural network, a convolutional neural network, an autoencoder, a deep learning architecture, a support vector machine, a data-driven trainable regression model, a k-nearest neighbor classifier, a physical model, and / or a decision tree, or a combination thereof. The machine learning model 613 may be trained on data received from one of several data sources 110.

[0076] The output 614 of the machine learning model 613 may be a relevance score for the data with respect to the feature and a confidence value in the accuracy of the relevance score. In the example of [Fig.6A], the relevance score is "TRUE", meaning that the image shows information relevant to "suitable for dogs" with the answer being "yes, this is suitable for dogs". This is because the photo shows a man with his dog on a kayak. The machine learning algorithm is also quite confident that this assessment is correct, indicated with a confidence value of 0.86 (= 86%).

[0077] [Fig.6B] shows an example of text content analysis after preprocessing. For each preprocessed observation as shown in box 620 with the preprocessed text 621 and the metadata 622, a machine learning model 613 may be applied. The input may be, e.g., at least one of, the preprocessed text, the words extracted from the text, at least a portion of the extracted metadata, and so on. The input is processed, e.g., by applying natural language processing to analyze the similarity of the words and / or the sentiments of the authors.

[0078] The machine learning model 623 may be a supervised or semi-supervised model. The machine learning model 623 may be one of a gradient boosted tree, a random forest, an artificial neural network, a recurrent neural network, a convolutional neural network, an autoencoder, a deep learning architecture, a support vector machine, a data-driven trainable regression model, a k-nearest neighbor classifier, a physical model, and / or a decision tree. The machine learning model 623 may be trained on data received from one of several data sources 110.

[0079] The output 624 of the machine learning model 623 may be a relevance score for the data with respect to the feature and a confidence value in the accuracy of the relevance score. In the example of [Fig.6B], the relevance score is “FALSE”, meaning that the image shows information relevant to “suitable for dogs” with the answer being “no, this is not suitable for dogs”. This is because the text indicates that the dog was not allowed for this activity. The machine learning algorithm is confident that this assessment is correct, indicated with a confidence value of 0.90 (= 90%). This high value may be due to the fact that “owner” is mentioned in the text, which was determined based on trained keywords.

[0080] In such travel-related examples, these machine learning models 613, 623 create the most sophisticated information possible. Therefore, they can aggregate information from multiple sources and multiple domains, learn on unlabeled data (and apply clustering or self-supervision algorithms), and integrate prior knowledge (e.g., Bayesian algorithms). Some typical tasks of these machine learning models 613, 623 are, e.g., sentiment analysis performed to evaluate users' perceptions of the activity and service around them or identification of interesting entities in images, or relevant actions performed in videos.In some embodiments, the machine learning models may replicate human analysis in a scalable manner, may lead to results that are not "computable" by humans, or . may lead to results that are computable by humans but with superhuman performance.

[0081] Although Figs. 6A and 6B only show one machine learning network to be applied on a preprocessed observation, the analysis may consist of a sequence of machine learning models or multiple machine learning models applied in parallel. Additionally, in some embodiments, the output of multiple machine learning models as well as other models may be combined to evaluate the relevance score and confidence value. The output of the analysis modules and processes is then used by the enrichment process of which [Fig. 7] is an example.

[0082] [Fig.7] depicts an enrichment process that uses a machine learning model 702 to determine whether the information should be used to enrich stored data regarding the selected feature. In this example, the input 701 consists of at least five values ​​for each of the N observations, namely, the relevance score and confidence value determined by the analysis, the confidence factor, the weighting factor, and the timestamp. In another example, more input values ​​may be selected, e.g., other values ​​extracted from the metadata.

[0083] N is an integer and indicates the number of observations received from one or more data sources. In some embodiments, N is predefined, meaning that the observations are grouped by the N observations. N may be chosen according to the available processing power. If more than N observations are received from the data sources, then the output of the enrichment processes may be combined, e.g., evaluated by forming average values ​​or the like. If fewer than N observations are received, the "missing" values ​​may be filled in with the average values ​​of the other observations.

[0084] The machine learning model 702 may be a supervised or semi-supervised model. The machine learning model 702 may be one of a gradient boosted tree, a random forest, an artificial neural network, a recurrent neural network, a convolutional neural network, an autoencoder, a deep learning architecture, a support vector machine, a data-driven trainable regression model, a k-nearest neighbor classifier, a physical model, and / or a decision tree, or a combination thereof. The machine learning model 702 may be trained on the preprocessed data.

[0085] The automatic learning model 702 can process the input (of size N x 5) to determine as output 703 what characteristic-related information can be added, e.g., in the example of [Fig.7], the information "TRUE", and a probability value of the accuracy of the information for data enrichment of data, e.g., in the example of [Fig.7] the probability value of 0.7 (= 70%). If the probability value is higher than a threshold value, the information is selected for data enrichment. For example, if the enrichment process determines that Kayak ABC is suitable for dogs (i.e., "true") with a probability of 70% and if the value of this threshold is set at 60%, the data stored in the database to be enriched are enriched with the information that Kayak ABC is suitable for dogs.

[0086] The interaction of all components is illustrated in [Fig. 8], which is a more fully detailed example of data enrichment. In this example, the first box 801 shows the data or observations, respectively, that are available in three data sources, namely website A, website B and the travel provider system. In addition, box 801 also already indicates the confidence factor that has been determined for each of the data sources. As can be seen in [Fig. 8], the confidence factor does not only depend on the data source as such but also on the data type (the text content of website A is evaluated with a confidence factor of 0.6, while the photo content of website A is evaluated with a confidence factor of 0.5). Box 801 also shows the raw data of each observation before preprocessing.

[0087] Box 802 then describes the result obtained after the preprocessing. For example, the structured data of the listing of Website B was transferred to a common structure. The structured data of the Travel Provider System was also brought into another structure, namely, a table comprising the relevant information (specifically here "Pet in cabin").

[0088] In Box 803, the output of the analysis is shown. For example, the second user review of Website A, which does not mention dogs at all, is indicated as not providing specific information. In some embodiments, this information will not serve as input for the enrichment process. This means that irrelevant observations will be filtered out so that less computing power will be required since less data has to be processed by the enrichment process.

[0089] Finally, box 804 shows what the result of the enrichment process is, namely, that kayak ABC is suitable for dogs with a probability of 65%. This probability may be too low so that no data enrichment in the stored data, e.g., stored in output storage 130 is performed. A high threshold ensures that end users are not provided with false information. In this embodiment, a human-readable explanation may also be provided so that a human supervisor will know why a data enrichment was performed or not.

[0090] As those skilled in the art will appreciate, there are no limitations on the features considered. For example, features regarding parking facilities may include parking difficulties, vehicle categories, height limit, indoor / outdoor parking, number of parking spaces, valets, public, price, amenities (electricity, water for mobile homes, etc.), crowded parking hours, time limits, distance to the supplier (Manhattan distance), and so on. Preprocessing and / or analysis processes may then include, for images, object detection (cars, borders, etc.), object segmentation (parking spaces, etc.)) and calculating weighted sums or other models, and for text, language translation, spelling correction, formatting, feature extraction, word similarity detection, and / or sentiment analysis on reviews. .

[0091] As another example, a feature about a good photo location may indicate whether there is a good photo to take for social media or whether it is a common photo location. In such an example, the preprocessing and / or analysis processes may take image metadata and cluster it to find the main points of attraction.

[0092] Features related to weather (rain, wind, etc.) and / or seasonality (winter, summer, etc.) may involve outdoor image detection and natural language processing (language translation, spelling correction, formatting, feature extraction...) such as preprocessing and natural language processing (word similarity, sentiment analysis on reviews...), hard-coded rules and word embeddings (kayak = outside?) as well as statistics on timestamps for analytics.

[0093] A feature may also concern a similarity of one point of interest to another. This can be achieved (in preprocessing or analytics) by applying Siamese networks on images, word embeddings on text reviews and other classical metrics on metadata.

[0094] Furthermore, the weighting factor may not only depend solely on the data type but also differ within a data type. For user reviews, the weighting factor may take into account the characteristics and behavior of the reviewer. A score may be calculated and it characterizes the neutrality of the reviewer. For example, a traveler who provides only negative reviews is less credible than a balanced traveler. The frequency of reviews may also be taken into account. For example, travelers who regularly provide feedback are given more weight than travelers who have submitted very few reviews. Additionally or alternatively, travelers who provide comprehensive / detailed reviews with images may be given more weight. Natural language processing and sentiment analysis may also be used to evaluate reviews provided by the same traveler.

[0095] [Fig. 9] shows a schematic representation of internal components of a computer system 900 implementing the functionality described herein. The computer system 900 includes at least a processor 901, a user interface 902, a network interface 903, and a main memory 906 that communicate with each other via the bus 905. Optionally, the computer system 900 may further include a static memory 907 and a hard disk drive (not shown) that also communicate with each other via the bus 905. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of the user interface 902. Furthermore, the computer system 900 may also include one or more graphics processing units (GPUs) 904.

[0096] The GPUs 904 may also include a plurality of GPU cores or streaming multiprocessors, which include different new components, such as at least one register, at least one cache and / or shared memory, and a plurality of ALUs, FPUs, tensor processing units (TPUs) or tensor cores, and / or other optional processing units.

[0097] The main memory 906 may be random access memory (RAM) and / or any other volatile memory. The main memory 906 may store program code for the functionalities as described herein. In particular, the main memory 906 may store data for machine learning models 908 that are applied to data analysis as well as machine learning model(s) 909 for enrichment processes as described herein. Other modules required for other functionalities described herein, such as data preprocessing and retrieval, may be stored in the memory 906 as well. The memory 906 may also store additional program data 910 required to provide the functionalities described herein.A portion of the program data 910 and / or the machine learning models 908, 909 may also be stored in a separate memory, e.g., in the cloud, and executed at least in part remotely. The main memory 906 may also include a cache 911 for storing, e.g., the output of the machine learning models 908 that are used as input for the machine learning model 909 or the like. .

[0098] According to one aspect, a computer program comprising instructions is provided. These instructions, when the program is executed by a computer, cause the computer to perform the methods described herein. The program code embodied in each of the systems described herein is capable of being individually or collectively distributed as a program product in various forms. In particular, the program code may be distributed using a computer-readable storage medium containing computer-readable program instructions for causing a processor to execute aspects of the embodiments described herein.

[0099] Computer-readable storage media that are inherently non-transitory may include volatile or non-volatile, removable and non-removable tangible media implemented in any information storage method or technology, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may further include random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory or other solid state memory technology, portable compact disc read only memory (CD-ROM) or other optical storage device, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other medium that can be used to store desired information and that can be read by a computer. .

[0100] A computer-readable storage medium should not be construed as transient signals per se (e.g., radio waves or other electromagnetic waves propagating, electromagnetic waves propagating through a transmission medium such as a waveguide, or electrical signals transmitted through a cable). Computer-readable program instructions may be downloaded to a computer, other type of programmable data processing apparatus, or other device from a computer-readable storage medium or to an external computer or external storage device via a network.

[0101] It should be noted that although specific embodiments and variants have been described herein, any modification and alternative will be apparent to those skilled in the art. In particular, the examples are provided by way of illustration of the principles and in order to provide a number of specific processes and arrangements for implementing these principles.

[0102] In certain embodiments, the functions and / or acts specified in the diagrams, sequential diagrams, and / or block diagrams may be reordered, processed serially and / or simultaneously without departing from the scope of the present disclosure. In addition, the diagrams, sequential diagrams, and / or block diagrams, may include more or fewer blocks than those illustrated according to the embodiment of the disclosure.

[0103] The terminology used herein is for the sole purpose of describing particular embodiments and is not intended to limit the embodiments of the disclosure. It is further understood that the terms "includes" and / or "comprising," when used in this specification, specify the presence of features, integers, steps, operations, elements, and / or components but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, to the extent that the terms "includes," "having," "has," "with," "consisting of," or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in the same manner as the term "comprising."

[0104] Although a description of various embodiments has illustrated the method and although these embodiments have been described in great detail, applicants do not intend to restrict or limit in any way the scope of the appended claims to such details. Other advantages and claims will be readily apparent to those skilled in the art. The disclosure in its broader aspects is therefore not limited to the specific details, the representative apparatus and method, or the illustrative examples shown and described. Accordingly, the described embodiments should be understood as being provided by way of example, for the purpose of teaching the general features and principles but should not be understood as limiting the scope which is as defined in the appended claims.

Claims

Claims

1. A computer-implemented method of enriching data according to a characteristic using data content from at least two data sources comprising at least one data type, wherein the method is executed in a computer system comprising a trust module, one or more pre-processing modules, one or more analysis modules and an enrichment module, the method comprising: • assigning, by the trust module, a trust factor to a data source for the characteristic; • pre-processing, by the one or more pre-processing modules, data received from the data source according to a data type; • analyzing, by the one or more analysis modules, information contained in the data concerning the characteristic;and • applying, by the enrichment module, an enrichment process to select the information for data enrichment related to the characteristic based on the analysis of the information and the confidence factor.;

2. The method of claim 1, wherein at least one data type comprises one or more of text, structured information, image, sound, video.

3. The method of claim 1, wherein preprocessing the data comprises extracting content from the data by at least one of: • for the text data type: applying natural language processing for translation, correction, formatting, voice labeling, and / or extracting features from the text; • for the structured information data type: organizing the content according to labels; • for the image data type: applying at least one of object detection, face detection, object segmentation, and object recognition; • for the sound data type: applying speech recognition and / or natural language processing; and • for the video data type: applying at least one of object detection, face detection, speech recognition, and natural language processing.

4. The method of claim 2 or claim 3, wherein analyzing the information comprises at least one of: • for the text data type: applying natural language processing to analyze the similarity of words and / or the sentiments of the authors; • for the image data type: analyzing detected objects and / or faces regarding at least one of facial expression, emotion, age, and gender; and • for the structured data type: discovering rules in the structure data.

5. The method of any one of the preceding claims, wherein the confidence factor is determined based on a machine learning model that is continuously trained with data retrieved from two or more data sources.

6. The method of any preceding claim, wherein preprocessing the data comprises extracting metadata from the data.

7. The method of claim 6, wherein, the metadata includes at least one of descriptive metadata regarding at least one of title, subject, genre, and author; rights metadata regarding at least one of title, copyright status, rights holder, and license terms; technical metadata regarding at least one of file type, size, creation date, creation time, compression type, uniform resource locator, and page ordering; preservation metadata regarding an article location in a hierarchy or sequence; and image metadata regarding at least one of a timestamp, camera properties, resolution, size, and geotag.

8. The method of any preceding claim, wherein analyzing the information comprises determining a relevance score for the data with respect to the feature and a confidence value in the accuracy of the relevance score.

9. The method of claim 8, wherein the enrichment process is based on a machine learning model, wherein the input to the machine learning model comprises the relevance score for the data with respect to the feature, the confidence value in the accuracy of the information, the confidence factor of the data source of the information, a weighting factor of the information, and a timestamp.

10. The method of any preceding claim, wherein the enrichment process for selecting the information for data enrichment comprises: • determining a probability value of accuracy of the information for data enrichment; and • in response to the probability value being higher than a threshold value, selecting the information.

11. The method of any preceding claim being triggered periodically and / or in response to a request for a request for recommendations regarding the characteristic.

12. A data enrichment system comprising a confidence module, one or more pre-processing modules, one or more analysis modules and an enrichment module configured to perform the method of any preceding claim.

13. A computer program product comprising program code instructions stored on at least one computer-readable medium for performing the steps of the method according to claims 1 to 11, when said program code instructions are executed on a computer.

Images