Event extraction system and method

Abstract

Events described in structured data (e.g., HTML web pages or emails) or in text described in natural language may be extracted and entered into one or more calendars on a user device. In one embodiment, selecting an add event command in a calendar application may cause the calendar application to search in a database of extracted events without receiving any search input, and may suggest events extracted within a predetermined period of time as events to be added to the calendar. In one embodiment, the extracted events may cause notifications to be displayed to the user. Other embodiments are also described herein.

Description

[0001] Cross - reference to related applications

[0002] This application is a divisional application of a patent application for invention, with international application number PCT / US2018 / 024182, international filing date of March 23, 2018, date of entry into the national phase in China of November 21, 2019, Chinese national application number 201880033649.X, and invention title "Event Extraction Systems and Methods".

[0003] This patent application claims priority to U.S. Patent Application 15,897,038 filed on February 14, 2018, U.S. Patent Application 15 / 897,043 filed on February 14, 2018, U.S. Patent Application 15 / 897,047 filed on February 14, 2018, and 15 / 897,053 filed on February 14, 2018, all of which claim priority to U.S. Provisional Patent Application 62 / 514,738 filed on June 2, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The various aspects and embodiments described herein relate to extracting events from different types of data. Background Art

[0005] Users of data processing systems typically send messages (e.g., text messages) or emails about events, such as going to a restaurant for dinner or lunch or going to a movie theater to watch a movie in the evening. Additionally, users of data processing systems typically use a web browser to book restaurants, car rentals, ticketed events (e.g., baseball games or movies, etc.), flights, hotels, etc.

[0006] In the past, data processing systems have been enhanced to include techniques for identifying different types of data such as events. See, for example, U.S. Patents 7,912,828, 8,423,288, and 8,738,360. These techniques rely on user interaction to utilize the extracted data. Summary of the Invention

[0007] In one embodiment, events can be extracted from natural language descriptions, such as certain types of emails or text messages or other text content. In another embodiment, events can be extracted from structured data such as HTML (e.g., web pages or certain types of emails, etc.).

[0008] A method for extracting events from a natural language description of an event may include the following operations: extracting an event from text having a natural language description; adding the extracted event to a database containing one or more extracted events; recording data representing a first time associated with the extracted event; displaying a user interface of a calendar application, where the user interface of the calendar application includes an add event command; receiving a selection of the add event command; in response to the selection, determining whether the first time associated with the event is within a time period of the current time; if the first time is within the time period of the current time, displaying at least a portion of the extracted event in the calendar user interface. In one embodiment, the most recently extracted event may be suggested as a new event to be added to the calendar in response to a command to add a new event. In one embodiment, the display of the extracted event may be a result of a search of an extracted event database, and the search result display may suggest auto-complete suggestions for events that may be extracted, suggesting events that may be extracted even if no characters have been entered into the search input field, and such auto-complete suggestions may be referred to as zero-word auto-complete suggestions. In other words, when zero characters are received in the search input field in the calendar user interface, at least a portion of the extracted event may be displayed in the calendar user interface. When the first time is outside the time period of the current time, it may be necessary to enter characters in the search input field in order to retrieve the extracted event as a possible search result in the auto-complete suggestion set of the search results. The natural language description may be part of a text message or an email.

[0009] In an alternative embodiment, when the user selects the add event command without considering the time associated with each extracted event, auto-complete suggestions from a search of the extracted events may be provided. In this alternative embodiment, these auto-complete suggestions obtained by searching the extracted event database may be provided without any characters being entered into the search input field (zero-word auto-complete suggestions) or after a character that matches an event in the extracted event database has been entered into the field.

[0010] In one embodiment, the first time associated with the extracted calendar event may be one of the following: (a) the time of receiving the text message or email; or (b) the time of extracting the extracted event when the text message or email is displayed; or (c) the time of adding the extracted event to the extracted event database.

[0011] In one embodiment, the method may further include determining a due date for the extracted event based on data extracted from the natural language description; and removing the extracted event from the database on or after the due date. In one embodiment, the database containing the extracted events may include a data structure in which the extracted events are sorted in order of time from most recent to least recent. In one embodiment, the method may further include receiving a selection of a displayed extracted event in a calendar user interface, and in response to the selection of the displayed extracted event, displaying a calendar event creation panel pre-populated with data from the extracted event to allow an entry to be edited into the calendar maintained by the calendar application based on the extracted event. In one embodiment, the extracted event may be placed in a sub-calendar, which may be characterized as a "found in application" calendar separate from the user's main calendar. In one embodiment, the calendar application may support multiple sub-calendars (such as a work calendar, a family calendar, etc.), as described in published U.S. patent application US2004 / 0109025.

[0012] Another embodiment of extracting events from a natural language description may include the following operations: extracting events from text having a natural language description; adding the extracted events to a database containing one or more extracted events; displaying a user interface of a calendar application, the user interface including a set of one or more dates in calendar format; receiving a selection of one of the one or more dates in calendar format; in response to the selection, searching the database for any extracted events for the selected date, the search being performed when zero characters are received in a search input field for receiving and initiating the search of the database; and displaying each of the extracted events in the database for the selected date as one or more candidate events. In one embodiment, the method may further include determining a due date for the extracted event based on data extracted from the natural language description; and removing the extracted event from the database on or after the due date.

[0013] Another aspect described herein relates to the extraction of one or more events when a user uses a web browser to make a reservation or otherwise create an event. In one embodiment, a method may include the operations of: receiving a document from a domain; comparing the domain to a set of domains in a domain whitelist; determining whether to continue processing the document based on at least one of a title of the document or a Uniform Resource Identifier (URI) of the document when the domain is in the set of domains; if processing continues, extracting structured data representing a candidate calendar event from the document; adding the candidate calendar event to a calendar database; and presenting a notification to the user that displays at least a portion of the data regarding the candidate calendar event. In one embodiment, the determination of whether to continue processing the document may be repeatedly made using different web pages of the same domain (or a domain known to be related to the original domain) during the process of creating an event while using a web browser. In one embodiment, the document is one of a web page, an email, or a text message that contains structured data from an enterprise. In one embodiment, the whitelist is a data structure in the memory of a process of the web browser, and the data structure is a memory-mapped lookup tree, and the comparison is a lookup operation in the memory-mapped lookup tree. In one embodiment, the determination of whether to continue processing may use a machine learning classifier that is trained on manually labeled examples for learning how to classify structured data based on a characterization of a title of a web page or a subject line of an email. In one embodiment, the characterization indicates whether an email or a web page is a confirmation of a reservation or an advertisement (non-event) or a promotion not considered an event. In one embodiment, the calendar database is a private local calendar database that is encrypted when stored in a user's private cloud storage account, and the candidate calendar event may be displayed in a sub-calendar, which may be one of a plurality of sub-calendars that are displayable in a calendar application.

[0014] In one embodiment, the notification may include at least one of the following: (a) a close command that exits the display of the notification while retaining the candidate calendar event; or (b) a select command that displays the candidate calendar event in a user interface of a calendar application, where the candidate calendar event is editable in the calendar application; or (c) a delete command that deletes the candidate calendar event from the calendar database. In one embodiment, the notification may be a consolidated notification that displays data regarding a set of seemingly related candidate events. In one embodiment, the method may further include removing duplicate events from the calendar database based on one or more of the following: a repetition count; or a repeated title; or a repeated source indicated by one or more domains. In one embodiment, duplicate events may be consolidated into the consolidated notification instead of being removed.

[0015] Another aspect described herein relates to an architecture for an extraction engine that may use different modules shared between different event categories. In one implementation, the extraction engine may perform the following operations: receive a document from a domain, the document including structured data such as HTML content; determine the language of the document; classify the document as either an event or a non-event; for an event described by the structured data, detect one or more of a location, an address, a date, a time, a phone number, or a uniform resource locator; select a data extractor from a set of data extractors for different event categories based on the domain determined to be in one of the categories; call a set of field extractors by the selected data extractor, each field extractor being configured to extract data from a corresponding type of field in the structured data; extract data within the fields of the structured data by the set of field extractors, wherein the extraction flow or order may be controlled by the selected data extractor; verify the data extracted from the fields; generate an output of the extracted data for the event; and automatically add the event to a calendar in response to generating the output upon successful verification. In one implementation, the set of data extractors includes data extractors for one or more of the following: restaurant reservations; car rental reservations; hotel reservations; ticketed events, including sports events and shows; flight reservations; or social invitation events. In one implementation, the method may further include comparing the domain to a domain whitelist; if the comparison shows that the domain is not in the whitelist, the method may stop before classifying the document. In one implementation, the method may further include preprocessing the document when the document is classified as an event to remove content not associated with the event before extracting data within the fields of the structured data. In one implementation, the classification determines whether the document is a cancellation of an event. In one implementation, each data extractor in the set of data extractors controls the data extraction flow of the field extractors when selected. In one implementation, the method may further include generating a key for the event to compare with other keys to remove duplicate events from the calendar. In one implementation, each field extractor in the set of field extractors extracts candidate objects from the document and scores the candidate objects to obtain the likelihood of being valid data for the fields of the event. In one implementation, the method may further include: if the verification is unsuccessful, sending a failure report to a server system, wherein the failure report includes the domain and one or more failure types. The failure report may be used to compare successful event extractions with failed event extractions and determine whether the event extraction software should be updated.

[0016] Another aspect described herein relates to the use of the extracted events in a map application that can display a map and can be used when traveling to a location to provide route guidance when traveling to an event. A method according to this aspect in one embodiment may include the following operations: extracting events from structured data or text having a natural language description, the extracted events including a location; adding the extracted events to a database containing one or more of the extracted events; receiving a selection to display the map application; in response to the selection to display the map application, searching the database for the extracted events including a location; displaying, within the map application, suggested options for displaying the locations of the extracted events. In one embodiment, the method may further include: filtering the results of the search by determining whether the location of the extracted event is within a threshold distance of the current location of the data processing system, and if the location of the extracted event is within the threshold distance of the current location, the extracted event is displayed as a suggested option. In one embodiment, the method may further include: filtering the results of the search by determining whether the time of the extracted event is within a predetermined time period of the current time, and if the time of the extracted event is within the predetermined time period of the current time, the extracted event is displayed as a suggested option. In one embodiment, if (a) other extracted events do not contain a location, or (b) the time of other extracted events is not within the predetermined time period of the current time, or (c) the location of other extracted events is not within the threshold distance of the current location, then the other extracted events in the extracted event database are not displayed as suggested options. In one embodiment, the search may be performed when no characters are entered into the search input field, and thus in one embodiment of the map application, the suggested options are zero-word autocomplete suggestions.

[0017] In one embodiment, the extracted event database has restricted access that requires the access rights possessed by the map application or a target daemon process, and other applications on the data processing system do not have access rights. In one embodiment, the map application may include a desktop applet accessible from the application launch user interface. In one embodiment, the location of the extracted event may be verified, and once verified, it may be geocoded to convert an address such as a street address into a set of geographic coordinates such as latitude and longitude. In one embodiment, the extracted event database may include the extracted events synchronized from one or more with a cloud storage account used by the data processing system to obtain the extracted events from other devices using the cloud storage account.

[0018] In one embodiment, the method may further include: displaying, within a user interface of a map application, a command for adding the extracted event to a calendar that can be displayed via a calendar application; adding the extracted event to the calendar application, and removing duplicate extracted events from at least one of a database or the calendar application. In one embodiment, the method may further include: determining a transportation mode; obtaining information about traffic or transportation delays from one or more server systems; determining an estimated arrival time based on the determined transportation mode and the information about traffic or transportation delays; and wherein the estimated arrival time is displayed together with suggested options within the map application.

[0019] The methods and systems described herein may be implemented by a data processing system such as one or more smart phones, tablets, desktop computers, laptop computers, smart watches, wearable devices, audio accessories, in-vehicle computers, and other data processing systems and other consumer electronic devices. The methods and systems described herein may also be implemented by one or more data processing systems executing executable computer program instructions stored on one or more non-transitory machine-readable media, which when executed cause the one or more data processing systems to perform one or more of the methods described herein. Accordingly, embodiments described herein may include methods, data processing systems, and non-transitory machine-readable media such as DRAM memory and flash memory.

[0020] The above summary does not include an exhaustive list of all embodiments of the present disclosure. All systems and methods may be practiced in accordance with all suitable combinations of the various aspects and embodiments outlined above and those disclosed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The invention is illustrated by way of example and is not limited to the figures of the various drawings, in which like reference numerals indicate like elements.

[0022] Figure 1 is a flowchart showing a method for extracting an event from a natural language description.

[0023] Figure 2A shows an example of a user interface of an instant messaging application that may include a natural language description of an event.

[0024] Figure 2B shows an example of a user interface of a calendar application.

[0025] Figure 2C shows an example of an autocomplete suggestion based on the extracted event, which may be displayed in the calendar application in response to a selection of a command for adding a new event to the calendar.

[0026] Figure 2D An example of a user interface of a calendar application is shown that can accept a search query entered into a search input field, and the search query can be used to search for extracted events in a database of extracted events.

[0027] Figure 2E An example of a user interface of an instant messaging application displaying a natural language description of an event cancellation is shown.

[0028] Figure 3A An example of a user interface of a calendar application displaying suggested events obtained by searching a database of extracted events is shown.

[0029] Figure 3B A method according to one embodiment is shown for displaying one or more candidate events in response to selection of a date in a calendar or in response to selection of a command to add a new event to a calendar.

[0030] Figure 4 An example in block diagram form of an architecture for using an event extractor with a calendar application is shown.

[0031] Figure 5 is a flow chart illustrating a method according to one embodiment in which one or more events may be extracted from structured data, such as a web page.

[0032] Figure 6A , Figure 6B and Figure 6C An example of web pages in a web browser during a browsing session as a user browses multiple web pages within a domain is shown, and a classifier determines, for each such page during the browsing session, whether to extract an event during the browsing session.

[0033] Figure 7 is available for execution Figure 5 Block diagram of the software architecture of the method shown in .

[0034] Figure 8A , Figure 8B , Figure 8C and Figure 8D shows that it can be used with, for example Figure 5 An example of using notifications together with the methods shown in .

[0035] Figure 9 is a flow chart illustrating a method of using an extraction engine that uses a hierarchical structure of components.

[0036] Figure 10 An example of an event extraction engine according to one embodiment is shown.

[0037] Figure 11shows an example of a hierarchical structure that can be used to implement the Figure 10 event extraction engine shown in

[0038] Figure 12A , Figure 12B , Figure 12C , Figure 12D , Figure 12E and Figure 12F show examples of various possible fields within six different event categories, including car rental reservations, ticketed entry events, restaurant reservations, hotel reservations, flight reservations, and social invitations.

[0039] Figure 13A , Figure 13B , Figure 13C , Figure 13D , Figure 13E , Figure 13F and Figure 13G show examples of validation rules that can be used as part of the processing performed by the Figure 10 event extraction engine shown in

[0040] Figure 14A is a flowchart that shows a method for improving event extraction software in one implementation by monitoring reports of successful and failed event extractions across a large number of devices.

[0041] Figure 14B shows a graph that shows how the Figure 14A method can trigger an update to the event extraction software, which can then be distributed to multiple devices to improve event extraction.

[0042] Figure 15 is a flowchart that shows a method for extracting events and using those extracted events to suggest locations in a map application in one implementation.

[0043] Figure 16 shows an example of a system that can be used with a map application to suggest locations within one or more of the extracted events.

[0044] Figure 17 shows an example of a user interface for a map application that can display suggested locations obtained from an extracted event database.

[0045] Figure 18 is a block diagram illustration of an example of a data processing system that can be used with one or more of the implementations described herein. DETAILED DESCRIPTION

[0046] Various embodiments and aspects will be described with reference to the details of the following discussion, and the accompanying drawings will illustrate the various embodiments. The following specification and drawings are exemplary and should not be construed as restrictive. Numerous specific details are described to provide a thorough understanding of the various embodiments. However, in some instances, well-known or conventional details are not described in order to provide a concise discussion of the embodiments.

[0047] As used in this specification, the term "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment. The phrase "in one embodiment" appearing in various places in this specification is not necessarily all referring to the same embodiment. The processes depicted in the following figures are performed by processing logic components that include hardware (e.g., circuits, dedicated logic components, etc.), software, or a combination of both. Although the processes are described below in a certain order of operations, it should be understood that some of the described operations can be performed in a different order. Additionally, some operations can be performed in parallel rather than sequentially.

[0048] One aspect described herein relates to extracting one or more events from text having a natural language description of one or more events. Figure 1 An example of a method using events extracted from text having a natural language description is shown. Figure 1 The method can use known data extraction techniques that extract data from natural language descriptions, where the data includes event data. Examples of such data extraction techniques are described in U.S. Patents 7,912,828 and 8,738,360. These known techniques can be used to perform operation 51 and can be used as Figure 4 part of the event extractor 251 shown in. Referring back to Figure 1 , in operation 51, one or more events are extracted from text having a natural language description. Figure 2A An example of a natural language description of an event displayed within the transcript 103 of the user interface of the instant messaging application 101 is shown. Referring back to Figure 1 , in operation 53, the extracted events can be added to a database containing one or more of the extracted events. Then, optional operation 55 can record data representing a first time associated with the extracted events. In one embodiment, the first time can later be used to determine whether the extracted events are presented as zero-word autocomplete suggestions. In another embodiment, the first time is not used, and autocomplete suggestions for the extracted events can be displayed without considering the time. In operation 57, a user interface of a calendar application can be displayed, and the user interface can include commands for adding events to the calendar. Examples of such commands are in Figure 2BAs shown, the figure shows a user interface of a calendar application, which includes an add event icon 121; the user can select the add event icon 121 to cause the calendar application to present a user interface such as Figure 2C the user interface shown in, to allow the user to add a new event to the calendar. Referring back to Figure 1 , in operation 59, the user interface of the calendar application can receive a selection of an add event command. The selection can include a touch or tap on the add event icon, or using a cursor and mouse to select the add event icon, or using other techniques known in the art to select the add event command represented by the add event icon. Then in Figure 1 operation 61 shown in, the system determines whether the first time is within a time period (e.g., one hour) of the current time in response to the selection received in operation 59; operation 61 will be performed in those embodiments that use the first time. If the first time is within the time period of the current time, the extracted event is displayed as a suggested event to be added to the calendar, and this is shown as operation 63 in Figure 1 . In one embodiment, the display of the extracted event can occur in response to a search of the extracted event database (where the user has not entered a search query), and all extracted events within a predetermined time period (e.g., one hour) of the current time can be presented as suggested events. In other words, in one embodiment, each such suggested event is a zero-word autocomplete suggestion that is displayed because the first time of each such event is within the time period of the current time. In one embodiment, the time period can be one hour, and in response to a selection of the add event command, all such extracted events extracted within the past hour can be presented as zero-word autocomplete suggestions of suggested events (discovered by a search engine).

[0049] The following example will illustrate the Figure 1 method by describing a text message session between two friends. Each friend can use an instant messaging application to exchange messages with the other friend in a session involving lunch. Figure 2A shows an example of the session on one device. The user interface 101 of the instant messaging application includes a transcript 103 that displays three message bubbles 105, 107, and 109. The user interface 101 also includes an input staging area 111 and a screen keyboard 112. The text input into the instant messaging application is analyzed by a hybrid engine that utilizes machine learning algorithms (which models are trained to recognize event intents in natural language descriptions) and an event knowledge database. Figure 4 The event extractor in is an example of such an event extractor 251 that analyzes the text in a natural language description, such as Figure 4The email 255 and message 253 shown in. Then, the event extractor 251 can extract events and store them in Figure 4 the database 257 shown in. Referring back to Figure 2A , it can be seen that the two friends agreed to have lunch at a restaurant called Fred's. In one embodiment, the event extraction system can assume a default time for lunch, such as 12:30 PM. The event extractor can extract the title of the event (such as "Lunch"), and can specify a default start time and default duration (if not explicitly mentioned in the natural language description).

[0050] The extracted events from the natural language description can be added to an extracted event database (such as database 257), and then the database can be searched in response to a selection of an add event icon (such as Figure 2B the add event icon 121 in the user interface 120 of the calendar application shown in. In other words, any user participating in Figure 2A the instant messaging session shown in can go to their calendar application to launch the user interface 120, and can select the add event icon 121. In response to the selection of the add event icon, a search of the extracted event database can be automatically performed to retrieve all recent events within a certain time period (such as one hour from the current time, where the user has not entered any search query). The search can be performed without selecting Figure 2B the search icon 122 in the user interface 120 of the calendar application shown in. In other words, the search performed in response to the selection of the add event icon 121 is automatically performed in response to the selection of the add event icon, without the need to separately select the search icon 122. The results of the search in response to the selection of the add event icon 121 are shown in Figure 2C , which represents zero-word autocomplete suggestions (searches from a search engine) of the extracted events that have been extracted in the past hour from the current time (in this embodiment). In one embodiment, the time period used in operation 61 is one hour, but it can be other time periods. As Figure 2C shown in, the extracted event 133 can be an autocomplete suggestion returned as part of the search results, where no characters have been entered into the search input field of the calendar application. The extracted event 133 shows that the event is considered a candidate event due to the word "possible", and includes the name and time of the restaurant and the invitee, who is Figure 2A another participant in the text messaging session shown in. If the user of the instant messaging application operating Figure 2A shown in has had other instant messaging sessions in the past hour, and those other messaging sessions have also created extracted events, then Figure 2CThe autocomplete suggestions for the extracted events shown in [e.g., from search engine 259] will also include those extracted events, since in one implementation these events were also extracted within the past hour.

[0051] Figure 1 The time-based method shown in [e.g.,] allows a user to retrieve all recently extracted events that were extracted from recent messages, emails, etc. and other natural language descriptions that the system has automatically processed without entering any characters into the search input field. Thus, the user does not need to enter any search query to retrieve recently extracted events, such as those extracted within the past hour in one implementation. It should be understood that in alternative implementations, other time periods other than one hour may also be used. In one implementation, the first time associated with the extracted event can be the time when the text message or email providing the natural language description of the extracted event was received; alternatively, the first time can be the time when the extracted event was extracted when the text message or email was displayed; alternatively, the first time can be the time when the extracted event was added to the extracted event database. In each case, the first time is compared with the current time to determine whether it is within the time period of the current time, such as within one hour of the current time. In one implementation, an event extractor (such as event extractor 251) can extract a due date or create a due date based on rules regarding a cut-off time and a text description; for example, by default, a lunch or restaurant reservation can be considered due on the day after the start date, so if the start date is Wednesday, August 17, the due date can be Thursday, August 18 by default. In one implementation, the due date can be used to remove one or more of the extracted events that have expired; in one implementation, all of the extracted events that have expired can be periodically removed from the extracted event database so that the search engine (e.g., search engine 259) does not return outdated extracted events in search results. The removal of the extracted events that have expired can also be performed in other implementations described herein, such as in implementations that extract events from structured data (e.g., Figure 7 , Figure 10 and Figure 16 the HTML content in a web page or email in the implementation shown in [e.g.,].

[0052] As Figure 2D shown in [e.g.,], the implementations described herein can also allow a user to search for the extracted events by entering one or more characters into a search input field 137 displayed within a user interface 120B of a calendar application. Entering one or more characters into the search input field 137 can cause Figure 4 the search engine 259 shown in [e.g.,] to search a database that contains the extracted events, which database is in Figure 4is shown as database 257 in. The results of the search can be provided by the search engine 259 to the calendar application 263, which can present a user interface as shown in Figure 2B , Figure 2C and Figure 2D as shown in.

[0053] In one embodiment, Figure 4 the event extractor 251 shown in can analyze the natural language description to determine whether the event has been cancelled. This can occur either before adding the event to the extracted event database (such as database 257), or after adding the event to the extracted event database. Detection of cancellation can cause the event extractor (such as event extractor 251) to remove the extracted event from the extracted event database (such as Figure 4 the database 257 shown in). Figure 2E shows an example of a conversation containing four speech bubbles 145, 147, 149 and 151. Speech bubbles 145, 147 and 149 indicate that an event (lunch) has been scheduled. However, speech bubble 151 states that the event has been cancelled because one of the participants is unable to attend the event, so this causes the event extractor (such as event extractor 251) to cancel the event, and this cancellation causes the event to be removed from the extracted event database (if the event has been added to the database).

[0054] Figure 3A and Figure 3B show another embodiment, where the extracted events can be returned as search results, and these search results can be autocomplete suggestions provided in the search results. Figure 3B The method shown in can start in operation 201, which extracts events from text having a natural language description. For example, operation 201 can use the event extractor 251 to extract events from a natural language description (such as the description found in the Figure 4 message 253 or email 255 shown in). Then, in Figure 3B operation 203, the extracted events can be added to a database containing one or more extracted events. This can be seen in Figure 4 where the event extractor 251 adds the extracted events to the database 257. Referring back to Figure 3B , the user can select or launch the calendar application, and in operation 205, the calendar application displays the user interface of the calendar application, which can include various different ways of displaying the calendar view (such as daily view, weekly view, monthly view, etc.). In one embodiment, the view can be a calendar format showing one or more dates, such as Figure 2BThe format shown in the figure, which shows calendar entries 125, 127, and 129 in the weekly view 123. Then in Figure 3B operation 207 of, the data processing system may receive a selection of a date in the calendar format displayed within the user interface of the calendar application. For example, the user may select the date shown in calendar entry 127 in Figure 2B . Then, Figure 3B the selection in operation 207 of may cause operation 209 to occur. In operation 209, the data processing system may search a database for any extracted events for the selected date in response to the selection of the date when zero characters have been entered into the search input field. Referring back to Figure 4 , the search engine 259 may search the database 257 for the extracted events for the selected date, and the results of the search may include one or more extracted events that may be displayed as one or more candidate events in Figure 3B operation 211 of. Figure 3A shows an example of a user interface 120C of the calendar application that may be generated by Figure 3B operation 211 of. In the example shown in Figure 3A , the extracted event 160 has been found by the search in Figure 3B operation 209 of, and the event is displayed in the user interface 120C. The candidate event is displayed as a candidate event due to the word "possible" in the extracted event 160, as shown in Figure 3A . The extracted event may have been created due to an email or text message between a user of the data processing system and another user who exchanges text in a natural language description that describes lunch at a restaurant called Mom's Burgers on the date selected in Figure 3B operation 207 of. If there are multiple extracted events for the selected date, the calendar user interface (such as user interface 120C) may display the multiple extracted events in a scrollable list. Then, a user of the data processing system that includes the calendar application may add the extracted event to the user's calendar by selecting the add event option 121A, or may not add the extracted event to the user's calendar by selecting the cancel option 156. The user interface 120C also allows the user to add a new event by selecting the new event option 157, in which case the new event will typically be different from the extracted event 160. In Figure 3BIn the method, it is not required that the user enter any characters in a search input field (such as search input field 137 in the user interface of a calendar application). Thus, the search results provided by the search engine can be regarded as zero-word autocomplete suggestions or a set of suggestions based on the selected date received in operation 207. It should be understood that the user can also search for the extracted events by entering characters in the search input field in the calendar application to retrieve the calendar events manually entered by the user as well as the extracted events in the extracted event database (such as database 257). Then, a search engine (such as Figure 4 the search engine 259 shown in

[0055] In Figure 3B an alternative embodiment of the method shown in Figure 3B operation 207 may receive a selection to add a new event instead of a selection of a date in the calendar application. For example, in a Figure 2B this alternative embodiment of operation 207 shown in

[0056] Figure 5 , Figure 6A , Figure 6B , Figure 6C , Figure 7 and Figures 8A to 8D show another aspect of the embodiments described herein, which can be used when processing structured data (such as a web page containing HTML content). According to the embodiments of this aspect, the method shown in Figure 5 can be used together with the architecture shown in Figure 7 , which can ultimately generate a set of one or more notifications, such as Figure 8A , Figure 8B , Figure 8C and Figure 8D the notifications shown in Figure 5 . Recalling Figure 7a set of domains (or other identifiers of the source) in the domain whitelist 403 shown in FIG. are compared. If the domain is not found in the whitelist, method 5 may stop at operation 303 and the method will not continue as long as the Web browser remains browsing within that domain. In one embodiment, the whitelist is a data structure in a memory-mapped lookup tree, and the comparison operation performed in Figure 5 operation 303 of FIG. may be a lookup operation in the memory-mapped lookup tree. If the domain of the document received in operation 301 is found in the whitelist in operation 303, the process continues to Figure 5 operation 305 shown in FIG. In one embodiment, when a user browses multiple web pages within the same domain (such as web pages 321, 323, and 325 respectively shown in Figure 6A , Figure 6B and Figure 6C ), Figure 7 the classifier 405 shown in FIG. may repeatedly execute operation 305 in a loop. In one embodiment, when a user is booking a flight or a restaurant or a movie or registering for other event categories, browsing multiple web pages within the same domain may occur, where the booking process requires the user to interact with multiple web pages within the domain. Referring back to Figure 5 , operation 305 determines whether to continue processing the document based on (in one embodiment) at least one of the title of the document or a uniform resource identifier such as the URL (Uniform Resource Locator) of the document. Operation 305 can be seen in Figure 7 , where the Web browser 401 provides the title and the URL 404 to the classifier 405 that performs Figure 5 operation 305 shown in FIG. When the user browses multiple web pages within the domain, operation 305 is repeated for each web page, and Figure 7 the Web browser 401 shown in FIG. repeatedly provides one or more titles and the URL 404 to the classifier 405, and then the classifier may indicate in decision 406 whether to extract an event from the web page. Decision 406 may indicate when to extract based on the classification of the web page provided by the classifier 405. In a typical example of the loop that operation 305 in Figure 5 may execute, when the classifier provides a "yes" decision 406 to the Web browser 401, the classifier may return a "no" decision 406 for a set of initial web pages before one or more sets of final web pages. This can be seen in Figure 6A , Figure 6B and Figure 6C . Specifically, the classifier 405 classifies the initial web page 321 as non-event. Similarly, at the initial stage of the booking process, the classifier 405 classifies the web page 323 as non-event. This can continue for multiple Web pages within the domain. At each instance of a new web page, Figure 7The Web browser 401 in provides the title and URL 404 to the classifier 405, which then classifies the web page with a "yes" decision (which is an event) or a "no" decision (which is a non-event and has not been extracted yet). In Figure 6C In the example shown, the web page 325 can be a web page that displays a confirmed reservation for an event, and the classifier 405 returns a "yes" decision 406 to the Figure 7 Web browser 401 in, which then provides the HTML content 408 to the Figure 7 Event extractor 407 shown in. Referring back to Figure 5 , the decision of the classifier 405 causes the processing to continue from operation 305 to operation 307, where structured data from a specific web page classified as an event by the classifier 405 is extracted. This structured data represents a candidate calendar event. After the structured data is extracted in operation 307 by, for example, the Figure 7 Event extractor 407 shown in, operation 309 shown in Figure 5 can be performed. In operation 309, the candidate calendar event can be added to a calendar database, which can be a database of sub-calendars, such as the sub-calendars described in published U.S. Patent Application 2004 / 0109025. For example, as shown in operation 309, the candidate calendar event can be added to a calendar characterized as "the calendar found in the application". This calendar database can be Figure 7 Part of the extracted event database 409 shown in. Then, adding the candidate calendar event can cause a notification system (such as the Figure 7 Notification system 411 shown in) to cause Figure 5 Operation 311 shown in. In operation 311, the data processing system can present a notification to the user, and the notification can display at least a portion of the data of the candidate event extracted in Figure 5 Operation 307 of. Referring back to Figure 7 , the notification system 411 can cause notifications (such as the Figure 8A , Figure 8B , Figure 8C And Figure 8D Shown in) to be displayed, and the user can interact with these notifications to cause the extracted event to be added to Figure 7 One or more calendar databases 412 shown in.

[0057] In one embodiment, the notification can be displayed to the user simultaneously when a predetermined process is completed, as shown in Figure 8A And Figure 8B Shown in. In Figure 8AIn the case of, when the notification 455 is displayed, the Web browser window 453 remains displayed on the screen 451 of the user's data processing system. The Web browser window 453 may display the final page of the reservation process, which may be a confirmation page providing details extracted by the Figure 7 event extractor 407 shown therein. Thus, in one embodiment, after the event extractor has extracted an event and added the extracted event to the extracted event database (such as the Figure 7 database 409 shown therein), the notification 455 may occur soon after the confirmation web page (such as the Figure 6C web page 325 shown therein) is displayed. In the case of the Figure 8B user interface shown therein, the Web browser window 457 is displayed on the screen 456, and multiple notifications 459 are displayed simultaneously based on multiple extracted events (such as a car rental reservation or a flight reservation) extracted from the reservation. Figure 8C FIG. shows an example of the notification 455, which includes at least a portion of the data of the candidate calendar event in the region 465, and also includes a close option 467 and a delete option 469. The user may select the region 465, which may cause the calendar application to open to allow the entry of the event to be edited into the user's calendar. If the user selects the close option 467, the notification is closed without adding the extracted event to the user's calendar; however, the extracted event may remain in the extracted event database (such as the extracted event database 409) until its deadline, which may be stored in a field as part of the data regarding a particular extracted event. If the user selects the delete option 469, the extracted event is not added to the user's calendar and is also deleted from the extracted event database (such as the Figure 7 extracted event database 409 shown therein). Figure 8D FIG. shows an example of a user interface of multiple notifications such as multiple notifications 459.

[0058] In one embodiment, Figure 7The classifier 405 shown in [figure] can be a machine learning classifier that is trained on manually labeled examples of web pages for understanding whether an email or a web page is a confirmation of a reservation or an advertisement or promotion of a non-event based on features of the title of the web page or the subject line of the email. In one implementation, events extracted by an event extractor (such as event extractor 407) can be added to a private local calendar database separate from the user's main calendar. If the private local calendar database is stored in the user's private storage cloud account, it can be encrypted. In one implementation, the private local calendar database containing the extracted events can be displayed in a sub-calendar, which is one of the multiple sub-calendars that can be displayed in a calendar application. In one implementation, notifications that seemingly pertain to the same event can be merged into a single notification, which can be displayed as Figure 8B and Figure 8D the notification 459 in [figure]. In one implementation, an identifier of an event (such as the following key) can be created to identify events belonging to a recurring event, thereby allowing removal of duplicate events from the calendar and the calendar database.

[0059] Another aspect of the implementations described herein relates to an event extraction engine that can use a component hierarchy, and within this hierarchy, there can be multiple data extractors, where each of the multiple data extractors is dedicated to a specific category of events (such as hotel reservations, car rental reservations, flight reservations, restaurant reservations, ticketed events, etc.). Figure 9 、 Figure 10 and Figure 11 show examples of this aspect. Additionally, the event extraction engine (such as event extraction engine 550) described in this aspect can be used in other aspects, such as for event extractor 407 and extraction system 755. In one implementation, Figure 10 the event extraction engine 550 shown in [figure] can execute Figure 9 the method shown in [figure], and the event extraction engine 550 can use Figure 11 the hierarchy 600 shown in [figure] to perform the event extraction process. Figure 9 The method shown in [figure] can start in operation 501, in which a document is received from a domain, and the document contains structured data such as HTML content. The document can be, for example, a web page or an email with structured data, or a text message with structured data provided by an extension application operating with an instant messaging application. Referring to Figure 10 , the document 551 is an example of the document received in Figure 9 operation 501 of [figure]. In one implementation, Figure 9 the method of [figure] can optionally use a domain whitelist, such as Figure 10The whitelist 553 of the domains (or other identifiers of the sources) shown in. In one embodiment, the optional use of the whitelist can be performed before the Figure 9 operation 503. If the domain is not found in the whitelist, then Figure 9 's method can stop at this point and no further processing of the current document will be continued. On the other hand, if the domain of the received document is found in the whitelist (such as Figure 10 's whitelist 553), then the processing can continue in operation 503. As shown by the operation 503 in Figure 9 , the classifier can classify the document as an event or a non-event for event extraction. In one embodiment, the structured data within the document can be provided to the Figure 10 shown machine learning classifier 557, which can be similar to the Figure 7 shown classifier 405. In one embodiment, the machine learning classifier 557 can be a classifier manually trained on labeled examples for learning how to classify emails or web pages containing structured data, and the classification can include classifying the structured data as a confirmation of a reservation or other event, and can also include the classification of non-events (such as advertisements and promotions, etc.). In one embodiment, the machine learning classifier can learn the characteristics of the subject line of a confirmation email and the subject line of an advertising email. For example, the machine learning classifier can learn that when the word "confirmation" appears in the subject line or title, this increases the likelihood that the email or web page has an event confirmation. On the contrary, the machine learning classifier can learn that when "10% discount" appears in the subject line or the title of the web page, the email or web page is less likely to contain an event reservation and may just be a commercial email or advertisement that should be discarded. For a web page, the classifier can use the title and the full URL of the web page for classification. In one embodiment, the classifier can also use statistical methods to determine the locale (language and country) of the email or web page. The determination of the language can also be performed by the Figure 10 shown conventional language recognizer 555, which in one embodiment provides the identification of the language to the Figure 10 shown machine learning classifier 557. This can allow the machine learning classifier to use appropriate rules for a specific language in one embodiment. In one embodiment, when the document is an email containing structured data, the machine learning classifier 557 can receive the subject line of the email and the email header. If the classifier (such as the machine learning classifier 557) classifies the document as an event, then the processing can continue at the Figure 9 operation 505. Operation 505 can include detecting one or more of the following from the structured data: location, date, time, phone number, address, or uniform resource locator or uniform resource identifier. The detection that can be performed in operation 505 can be by Figure 10is performed by the data detector 561 shown in. In one embodiment, the data detector 561 can be a set of known data detectors for detecting different types of data such as location, date, time, phone number, address, uniform resource locator, etc. In one embodiment, each data detector in the set of data detectors can be dedicated to detecting only a specific type of structured data such as location, date, time, etc. In one embodiment, the data detector 561 can perform operation 505 to remove non-event content after an optional pre-processor such as the pre-processor 559 is used to pre-process the structured data. This is shown in Figure 10 because the pre-processor 559 is located before the data detector 561 in the data flow shown within the event extraction engine 550. In one embodiment, the pre-processor 559 removes irrelevant HTML comments. For example, the pre-processor 559 converts " hello " to "hello", and " <a href＝"http: apple.com”alt＝"apple”>”Convert to "<http: / / apple.com>" etc.

[0060] After detecting the data of the structured data from the document in operation 505, the process can continue to Figure 9 In operation 507. In operation 507, the event extraction engine can select a specific data extractor from a set of data extractors for different event categories through a category selector such as the Figure 10 category selector 563 shown in. This selection can be made based on the domain determined to be in a category. For example, if an email is received from "OpenTable.com" or the domain of a web page is "OpenTable.com", then the category selector 563 will select a data extractor for restaurant reservations, such as the Figure 11 data extractor 609 for restaurant reservations shown in. In one embodiment, the Figure 10 category selector 563 shown in can select one of the Figure 11 data extractors shown in, such as data extractors 603, 605, 607, and 609. In one embodiment, there can be six such data extractors, which are selected by the Figure 10 category selector 563 shown in. These six categories are in Figure 12A , Figure 12B , Figure 12C , Figure 12D , Figure 12E and Figure 12Fis shown in. In one embodiment, the category selector 563 may use a data structure that includes a list of domains in each of these six categories. The category selector 563 may perform a lookup in the list of domains and determine the category from the category specified for the domain. For example, there may be a list of domains for car rental reservations, a list of domains for restaurant reservations, a list of domains for flight reservations, a list of domains for hotel reservations, a list of domains for ticketed events, etc. In one embodiment, each such category has a separate data extractor, and this group of such data extractors is shown as a group of data extractors 565 in Figure 10 . The role of each data extractor is to control the general flow of that category. For example, the restaurant data extractor will attempt to extract the date and time of the event and the location (if available), and then it can call specialized reusable sub-modules or field extractors that handle very specific types of extraction in specific fields. This is shown in operation 509, where the selected data extractor calls a group of field extractors for each field expected to be extracted for the event category. The group of field extractors can be the Figure 10 field extractors 567 shown in or Figure 11 a group of field extractors 611 shown in. In one embodiment, for each field shown in Figures 12A to 12F , there are separate and distinct field extractors for the specific field, although these field extractors may be shared among different categories. For example, the field extractor for the "start date" field may be shared by the ticketed event category, the restaurant reservation category, the hotel reservation category, and the flight reservation category, etc.

[0061] Referring back to Figure 9 , the group of field extractors called by the selected data extractor may extract the data within the fields of the structured data in operation 511. The selected data extractor delegates the extraction of all the required attributes of the event in its category, and then puts these details together to create an event, which can be verified in operation 513 shown in Figure 10 by the validator 569 shown in Figure 9 . In one embodiment, the sub-module or field extractor may collect a set of candidates for extracting data in the document and score them using various criteria so that only the highest-scoring items are retained as the extracted data for a specific field. For example, the start time field extractor may obtain all the time candidates extracted using the data detector 561 and give a higher score to the candidates close to keywords such as "start time" or "your reservation time is" or "starts at", etc. After the selected data extractor combines the events extracted, it can use, for example, Figure 13B , Figure 13C , Figure 13D , Figure 13E ,Figure 13F and Figure 13G the validation rules for the specific categories shown in (and all categories in the case of Figure 13A ) validate the data. For example, in one embodiment, if a category selector (such as the category selector 563 in Figure 10 or the category selector 601 in Figure 11 ) selects the restaurant reservation category, the validation rules will include the rules shown in ​ and ​ . It should be understood that there are various alternatives for the validation rules, so alternative validation rules can be used in alternative embodiments, which may have fewer rules, more rules, or different rules in various categories. Referring back to ​ , in one embodiment, after successfully completing the validation operation (in operation 513), an event can be added to the calendar database, such as the "found in the application" calendar. In one embodiment, before adding the extracted event to the database of the extracted events, the extracted data can be processed to generate a formatted output, which can be a dictionary representation of the event, such as event details that can follow, for example, the schema.org Web standard for various categories. In other words, the standard for formatting a specific type of event can be used to generate a formatted output according to the standard. In addition, the process of generating the output in a specific format can also include generating a title for the event and metadata about the event such as the duration of the time, where if the event does not provide a duration, the duration is the default duration. In one embodiment, when the event is non-explicit, the default duration of the event can be inferred; for example, the default duration of a restaurant reservation is 1 1 / 2 hours. The process of generating the output of the extracted event can also include generating a duplicate keyword for checking whether a similar event already exists in the user's calendar, and when a duplicate event has been entered into the calendar, this duplicate keyword can be used to remove duplicates from the calendar. For example, in the case of a restaurant reservation, the duplicate keyword can be a keyword such as restaurant / reservation identifier / name / time. This keyword can also be used when canceling the extraction of an event so that the cancellation can be used to remove the canceled event. Once an event is extracted (such as the extracted event 571 shown in ​ ), it can be added to the database of the extracted events, such as the database of the extracted events 573 shown in ​ . In one embodiment, adding the extracted event to the database of the extracted events can automatically populate the "found in the application" calendar with the extracted event so that the extracted event can be found by a search engine, which can be invoked by the user to search for events within the calendar application or can be invoked by the system in response to an event such as the user selecting an add event command within the calendar application.

[0062] As described herein, the event extraction engine 550 may employ a hierarchical structure of components, such as the hierarchical structure 600 shown in ​ . The category selector 601 is located at the top of the hierarchical structure and is responsible for selecting a specific data extractor for a particular category selected by the category selector. In one embodiment, a category selector 601 similar to the category selector 563 in ​ may select a category based on the domain that is the source of a document, such as document 551. The category selector 601 may then select a specific data extractor designed to work with the selected category. Subsequently, the selected data extractor (which may be one of the data extractors 603, 605, 607, and 609 shown in ​ ) and other data extractors may invoke an appropriate set of field extractors within a set of field extractors 611 according to the specific category. ​ , ​ , ​ , ​ , ​ and ​ illustrate various field extractors for different categories in one embodiment. The selected data extractor will select an appropriate field extractor according to the category used to process data as described herein.

[0063] Another aspect of the embodiments described herein is shown in ​ and ​ . Generally, a provider of a subscription service may change its web page format or its email format. This may cause the event extraction engine to no longer be able to successfully extract events. In some embodiments, these situations may be avoided by monitoring the success and failure of event extraction on multiple devices used by a user over a period of time. In one embodiment, this may involve tracking the success and failure of all event categories (such as the categories shown in ​ ) for all providers in all languages. ​Shows an example of a method that can provide such tracking. In operation 651, reports of successful event extractions can be received from multiple client devices, and each report can include an identifier (or other source identifier) of the domain from which the event was successfully extracted. The report can also include a set of metadata about the event and a document containing the structured data from which the event was successfully extracted. In one embodiment, in addition to the identifier of the domain, the report can also include a date indicating the extraction date. In operation 653, reports of failed event extractions can be received from multiple devices on a given date. Each report can include the identifier of the domain (or other source identifier) and one or more identifiers of one or more fields for which the extraction failed. These identifiers can be considered as error messages that indicate the type of error for a given domain on a given date. In one embodiment, these metrics can be transmitted daily by multiple client devices and collected in a privacy-preserving manner (such as without user identifiers, etc.) on one or more server systems and can be used to establish a baseline. A baseline can be established, for example, by operation 655, which compares the number of successful event extractions within a given time period with the number of failed event extractions within the same time period and displays this comparison over time. In one embodiment, the display can be similar to ​ the graph 670 shown therein, which tracks over time the successful event extractions 673 within a given time period relative to the failed event extractions 671 for the same domain. ​ The Y-axis shown in

[0064] ​ shows the number of reports of event extractions for a particular domain. It can be seen that at time 675 (time T1), the provider at domain "ABC" may have changed the format in the web page or email, which affected the accuracy of the event extraction process for one or more fields from the provider at domain "ABC". This comparison can show a high deviation from the baseline, which indicates that the event extraction engine may need to be modified. In one embodiment, this can involve obtaining sample documents from the affected domain (such as domain "ABC"). For example, this can be done by going to the website of the affected domain and making a reservation (and then canceling) and collecting sample documents (such as email confirmation documents and web page confirmation documents). Then, these sample documents can be run through the event extraction engine to allow modification of the event extraction engine to allow correct processing of the sample documents to successfully extract events. Once the event extraction software is modified based on this test, it can be transmitted to multiple client devices of the user in operation 657. ​ and ​ shows another aspect of the embodiments described herein, which relates to an automatic search in response to, for example, a selection made by a user to display a map application. ​ The method shown in ​used in conjunction with the system shown in to generate ​ the user interface 801 of the map application shown in. Now refer to ​ , in operation 701, events can be extracted from structured data or text with natural language descriptions, where the event includes a location. In operation 703, the extracted events can be added to a database containing one or more extracted events. Now refer to ​ , the system 750 can receive source material 753 that can be processed by the extraction system 755 to generate the extracted events that are added to a database 757 containing one or more extracted events. Thus, the extraction system 755 can process the source material 753 to perform ​ the operations 701 and 703 shown in. In one embodiment, the source material can include web pages, emails, text messages, etc. In one embodiment, the extraction system 755 can be similar to ​ the event extractor 251 shown in or similar to ​ the event extractor 407 shown in or similar to ​ the event extraction engine 550 shown in. In one embodiment, the extracted events can be added to a calendar application, such as ​ the calendar application 759 shown in. The extracted events can be continuously added to the calendar application as described herein, either automatically or after a notification that the user can manually confirm, or in response to the user's selection of a command displayed within the user interface of the map application, etc. Referring back to ​ , at some point, the user can launch or open the map application, which is shown in operation 705, in which the system receives a selection to cause the map application (or a desktop applet of the map application) to be displayed. When the user opens or launches the map application, the map application (such as ​ the map application 751 shown in) can query the target daemon (such as ​ the target daemon 761 shown in) to cause a search for the extracted events to be performed in the extracted event database (such as database 757). This search is shown in ​is shown as operation 707, and this search can be automatically performed in response to a selection to open or display a map application. In one embodiment, the search is restricted or filtered to only include the extracted events, and these extracted events include the verified locations. In one embodiment, this search can be performed even if no search input characters are entered into the search input field in the map application. In one embodiment, the search results can be filtered based on the current location or the current time or both of these parameters. For example, events that are not within the next eight hours (for flights) and the next four hours (for any other events found in the application or other events included in the extracted event database) will be ignored or filtered out by the search process. In one embodiment, events that do not have a structured geocoded location (such as latitude and longitude) are also ignored. In one embodiment, events that have a location more than 200 miles away from the current location are also ignored. In one embodiment, in ​ in the system 750 shown, the current time and the current location can be provided by the clock 763 and the location determination system 767. In one embodiment, the location determination system 767 can include one or more of a GPS system or an assisted GPS system or Wi-Fi positioning technology, etc. Thus, the automatic search performed by ​ operation 707 can be filtered based on the current location and the current time so that only events within, for example, four hours of the current time and within 200 miles of the current location are shown as suggestions in operation 709 in one embodiment. The suggestion options can be considered as auto-complete suggestions based on no characters being entered into the search input field. In one embodiment, if there are no extracted events within a predetermined time period, there will be no suggestion options shown in operation 709. In one embodiment, the shown suggestion options can include the estimated time of arrival (ETA) based on the current traffic conditions obtained from a map or traffic server (such as ​ the map and traffic server 765 shown).

[0065] ​ shows ​ an example of the output of operation 709. ​ The user interface 801 of the map application shown in ​The user interface 801 shown in. The suggested option 807 may include the name of the event and the name of the restaurant in this example, as well as the time and date of the event. In ​ In the example shown, the suggested option 807 shows that there is a reservation extracted from a source document (such as source document 753). The reservation is at Mom's Burgers at 7:00 tonight. Referring back to ​ , the user in operation 711 can select a suggested option, which can then cause the display of ​ A map of the location of the extracted event shown in, where the X marks the location of Mom's Burgers. In one embodiment, ​ The events extracted and stored by the system shown in can be ​ Any of the categories shown. In one embodiment, the database 757 can enforce restricted access, which requires access rights or permissions from, for example, the target daemon 761, where other daemons and other applications will not be able to access the database 757 because they do not have the access rights or permissions controlled by the database 757. In one embodiment, ​ The extraction system 755 shown in can include a method for geocoding an address by converting the address into a set of geographical coordinates as part of the verification process. In one embodiment, the database 757 can include the extracted events synchronized from one or more cloud storage accounts, and the cloud storage account is used by the ​ The system shown in to obtain the extracted events from other devices using the cloud storage account. In one embodiment, the suggested option 807 can include an estimated arrival time exported from data provided by a traffic server (such as the traffic server 765 shown in ​ ), where the traffic data is obtained via a network interface such as a cellular phone connection or a Wi-Fi connection. In one embodiment, the map application can display commands that the user can select, and when selected, the extracted events are added to the calendar provided by the calendar application.

[0066] The systems and methods described herein can be implemented in a variety of different data processing systems and devices, including general-purpose computer systems, special-purpose computer systems, or a combination of general-purpose and special-purpose computer systems. Exemplary data processing systems that can use any of the methods described herein include server systems, desktop computers, laptop computers, tablets, smartphones, cellular phones, personal digital assistants (PDAs), embedded electronic devices, or other consumer electronic devices.

[0067] ​ Is a block diagram of the hardware of a data processing system according to one embodiment. Note that although​ illustrates various components of a data processing system that can be incorporated into a mobile device, a handheld device, or other electronic devices, but this is not intended to represent any particular architecture or manner of interconnecting these components, as such details are not closely related to the present invention. It should also be understood that other types of data processing systems with fewer or more components than those shown can be used in conjunction with the present invention. ​

[0068] As ​ shown, the data processing system includes one or more buses 1309 for interconnecting the various components of the system. One or more processors 1303 are coupled to one or more buses 1309 as is well known in the art. The memory 1305 can be DRAM or non-volatile RAM, or it can be flash memory or other types of memory or a combination of such memory devices. The memory is coupled to one or more buses 1309 using techniques known in the art. The data processing system may also include non-volatile memory 1307, which can be a hard disk drive or flash memory, or a magneto-optical drive or magnetic memory, or an optical drive or other types of memory systems that maintain data even after the system is powered off (e.g., ROM). Both the non-volatile memory 1307 and the memory 1305 are coupled to one or more buses 1309 using known interface and connection techniques. A display controller 1322 is coupled to one or more buses 1309 to receive display data to be displayed on a display device 1323. The display device 1323 may include an integrated touch input to provide a touch screen. The data processing system may also include one or more input / output (I / O) controllers 1315 that provide an interface for one or more I / O devices, the one or more I / O devices being such as one or more mice, touch screens, touch pads, joysticks, and other input devices (including those known in the art), as well as output devices (e.g., speakers). Input / output devices 1317 are coupled through one or more I / O controllers 1315 as is well known in the art.

[0069] Although ​The non-volatile memory 1307 and the memory 1305 are shown as being directly coupled to one or more buses rather than through a network interface. However, it should be understood that the present invention may utilize non-volatile memory remote from the system, such as a network storage device coupled to the data processing system through a network interface such as a modem or an Ethernet interface. As is well known in the art, the buses 1309 may be connected to each other through various bridges, controllers, and / or adapters. In one embodiment, the I / O controller 1315 includes one or more of a USB (Universal Serial Bus) adapter for controlling USB peripheral devices, an IEEE 1394 controller for IEEE 1394-compatible peripheral devices, or a Thunderbolt controller for controlling Thunderbolt peripheral devices. In one embodiment, one or more network devices 1325 may be coupled to one or more buses 1309. One or more network devices 1325 may be wired network devices (e.g., Ethernet) or wireless network devices (e.g., Wi-Fi, Bluetooth).

[0070] It will be apparent from this description that aspects of the present invention may be embodied, at least in part, in software. That is, these techniques may be implemented in a data processing system in response to its processor executing a sequence of instructions contained in a storage medium, such as a non-transitory machine-readable storage medium (such as volatile DRAM or non-volatile flash memory). In various embodiments, hardwired circuitry may be used in combination with software instructions to implement the present invention. Thus, these techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source of the instructions executed by the data processing system. Additionally, it should be understood that in the case of describing a mobile device or a handheld device, such description encompasses mobile devices (e.g., laptop computer devices, tablet devices), speaker systems with integrated computing capabilities, handheld devices (e.g., smart phones), and embedded systems suitable for use in wearable electronic devices.

[0071] The present disclosure recognizes that in the techniques of the present invention, the use of personal information data (such as the extracted events that may be added to the user's calendar) can be used to benefit the user. For example, the personal information data can be used to automatically populate the user's calendar with the extracted events. Additionally, the present disclosure also contemplates other uses of the personal information data that are beneficial to the user.

[0072] The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Additionally, such collection should only occur after the user's informed consent. Further, such entities should take any steps required to safeguard and protect access to such personal information data and ensure that others who have access to the personal information data comply with their privacy policies and procedures. Additionally, such an entity may subject itself to third-party assessments to demonstrate its compliance with widely accepted privacy policies and practices.

[0073] Notwithstanding the foregoing, the present disclosure also anticipates embodiments in which users selectively block the use or access of personal information data. That is, the present disclosure anticipates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the techniques of the present invention may be configured to allow a user to select to "opt-in" or "opt-out" of participating in the collection of personal information data during a registration service. In another example, a user may select to "partially opt-in," where data is only collected when certain applications are used, or events extracted are added to a private encrypted calendar, etc.

[0074] In the foregoing specification, specific exemplary embodiments have been described. It will be apparent that various modifications may be made to those embodiments without departing from the broader spirit and scope given by the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a limiting sense.

Claims

1. A method, comprising: Receiving a document from a network domain, the document containing structured data; Classifying the document as corresponding to either an event or a non - event; Determining, based on the domain name of the network domain, an event category corresponding to the network domain, wherein the event category is one of a plurality of different event categories; Selecting a data extractor from a set of data extractors for the plurality of different event categories; Invoking, by the selected data extractor, a set of field extractors, each of the field extractors being configured to extract data from a corresponding type of field in the structured data; Extracting, by the set of field extractors, data within the fields of the structured data, wherein the extracted data includes at least one candidate start time and one or more of a confirmed natural language description or a reservation identifier; Combining, by the data extractor, the extracted data into an extracted event, wherein the extracted event includes one of the at least one candidate start time as the start time based on a keyword in the document whose start time is close to the start time described for the extracted event; Validating the extracted event using validation rules for the event category, wherein the validation is successful when the extracted event includes the start time and one or more of the confirmed natural language description or the reservation identifier; Displaying, via a display, a visual indication of an output representing the extracted event; And Adding the output representing the extracted event to an extracted event database to automatically populate a calendar with the extracted event.

2. The method according to claim 1, wherein the set of data extractors includes data extractors for one or more of the following: restaurant reservation; car rental reservation; hotel reservation; ticket - based entry events, including sports events and performances; flight reservation; or social invitation events; and wherein if the validation confirms that the extracted data represents an event, the event is added.

3. The method according to claim 2, wherein the method further comprises: Determining the language of the document; Comparing the domain name with a domain whitelist, and wherein if the domain name is not in the whitelist, the method stops before classifying the document.

4. The method according to claim 3, wherein the structured data describes the event, and wherein the method further comprises: Detecting one or more of the following for the event described by the structured data: location; date; time; telephone number; Or uniform resource locator; If the document is classified as corresponding to the event, processing the document to remove non - event content not associated with the event before extracting data within the fields of the structured data, wherein the document contains content in HTML format, and wherein the non - event content includes HTML comments.

5. The method according to claim 1, wherein the classification determines whether the document is a cancellation of the event.

6. The method according to claim 1, wherein each data extractor in the set of data extractors controls a data extraction stream when selected.

7. The method according to claim 1, wherein the plurality of different event categories includes two or more of the following: car rental reservation category, ticketed entry category, restaurant reservation category, hotel reservation category, or flight reservation category, and wherein the verification is successful when: the extracted data includes a start date and one of the following when the data extractor is used for the car rental reservation category, the extracted data includes a provider and a return date or a pick-up address; when the data extractor is used for the ticketed entry event category, the extracted data includes an event name and a ticket number; when the data extractor is used for the restaurant reservation event category, the extracted data includes a positive number of party members; when the data extractor is used for the hotel reservation event category, the extracted data includes a check-in time; when the data extractor is used for the flight reservation event category, the extracted data includes an airline IATA code, a flight number, a departure time and an arrival time, and departure and arrival airport codes or airport names.

8. The method according to claim 1, wherein the method further comprises: generating a key for the event to compare with other keys to remove duplicate events in the calendar.

9. The method according to claim 1, wherein each field extractor in the set of field extractors extracts candidate objects from the document and scores the candidate objects to obtain the likelihood of valid data for the fields of the event, and wherein the structured data is in HTML format.

10. The method according to claim 1, wherein the method further comprises: if the verification is not successful, sending a failure report to the server system, wherein the failure report includes the domain name and one or more failure types.

11. The method according to claim 1, further comprising: in response to the domain name of the network domain being in a whitelist domain, wherein the whitelist domain is a data structure in a memory-mapped lookup tree.

12. A non-transitory machine-readable medium storing executable instructions that, when executed by a data processing system, cause the data processing system to perform the method according to any one of claims 1-11.

13. An electronic device that communicates with one or more input devices and a display, the electronic device including one or more processors; A memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for the method according to any one of claims 1-11.

Images