Anaphora resolution method, system, and program

Abstract

A method and system for resolving anaphora includes extracting personal context data from a personal usage, determining whether the personal usage includes an anaphoric expression based on the personal context data, extracting anaphoric context data based on determining that the personal usage includes an anaphoric expression, and identifying an object of one or more objects to which the anaphoric expression refers based on a comparison of the personal context data and the anaphoric context data with data detailing the one or more objects.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Exemplary embodiments relate generally to text mining, and more particularly to contextual text mining. [Background technology]

[0002] Anaphora is a word that refers to or replaces a word used earlier in a sentence. For example, in the sentence "I want this", the pronoun "this" is an anaphora that refers to an object. Techniques for analyzing and determining the object to which such a pronoun refers are known as anaphora resolution. While anaphora is commonly used and understood in natural language, the growing popularity of chatbots, virtual assistants, and other human-computer interfaces has created a need for techniques that can quickly and effectively resolve anaphora in computer-based systems, especially when user input data is limited.

[0003] Under traditional anaphora resolution systems, anaphora resolution relies primarily on analysis of speech across a dialogue. For example, U.S. Pat. No. 9,754,591 B1 describes a method for resolving anaphora, where a user can converse with a computing device to initiate some function or obtain some information. A dialog manager can manage the conversations and store contextual data about one or more of the conversations. Processing and responses to subsequent conversations can benefit from the previously stored contextual data, for example by reducing the amount of information that a user must provide if the user has already provided information in the context of a previous conversation.

[0004] Improvements to the voice-based techniques used in resolving anaphora have been contemplated, for example, by collecting additional information about the anaphoric expression. For example, U.S. Pat. No. 9,767,501 B1 describes a method for resolving anaphora in which a user can provide voice input via a microphone and activate a scanner to scan an item identifier (e.g., a barcode). The handheld device can communicate the voice data and item identifier information to a remote system for voice-assisted scanning. The remote system can perform automatic speech recognition (ASR) operations on the voice data and can perform item identification operations based on the scanned identifier. Natural language understanding (NLU) processing can be improved by combining the ASR information with item information obtained based on the scanned identifier.

[0005] Further conventional approaches to resolving anaphora have been contemplated, and overall there have been various conventional approaches to improving anaphoric resolution, however each of these conventional approaches fails to take into account other contextual data regarding the anaphoric expression. Summary of the Invention

[0006] Exemplary embodiments disclose methods, configurations, and computer systems for resolving anaphora. Exemplary embodiments may include extracting individual context data from an individual expression, and determining whether the individual expression includes an anaphoric expression based on the individual context data. Exemplary embodiments may further include extracting the anaphoric context data based on determining that the individual expression includes an anaphoric expression, and identifying an object of the one or more objects to which the anaphoric expression refers based on a comparison of the individual context data and the anaphoric context data to data detailing the one or more objects.

[0007] The following detailed description, given by way of example and not intended to limit the illustrative embodiments thereto, is best understood in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0008] [Figure 1] 1 is an exemplary schematic diagram of an anaphora resolution system 100, in accordance with an exemplary embodiment. [Diagram 2] 2 is an exemplary flowchart 200 illustrating the operation of the anaphora resolver 132 of the anaphora resolution system 100, according to an exemplary embodiment. [Diagram 3] 1 is an exemplary schematic diagram of an environment 300 in which the anaphora resolution system 100 is implemented, according to an exemplary embodiment. [Figure 4] 4 is an exemplary table 400 representing data stored in an object database server 120 of an anaphora resolution system 100, according to an exemplary embodiment. [Diagram 5] 2 is an exemplary block diagram illustrating hardware components of the anaphora resolution system 100 of FIG. 1 in accordance with an exemplary embodiment. [Figure 6] FIG. 1 illustrates a cloud computing environment in accordance with an exemplary embodiment. [Figure 7] FIG. 1 illustrates an abstraction model layer in accordance with an illustrative embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The drawings are not necessarily to scale. The drawings are merely schematic and are not intended to depict specific parameters of example embodiments. The drawings are intended to depict typical example embodiments only. In the drawings, like numbering represents like elements.

[0010] Detailed embodiments of the claimed compositions and methods are disclosed herein. However, it can be understood that the disclosed embodiments are merely examples of the claimed compositions and methods, which may be embodied in various forms. The exemplary embodiments are merely illustrative, but may be embodied in many different forms, and should not be construed as being limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope covered by the exemplary embodiments to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0011] References herein to "one embodiment," "an embodiment," "an exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, configuration, or characteristic, but that all embodiments may not necessarily include the particular feature, configuration, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, configuration, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one of ordinary skill in the art to implement such feature, configuration, or characteristic in connection with other embodiments, whether or not explicitly described.

[0012] In the following detailed description, some process steps or operations known in the art are combined together for presentation and illustration purposes and in some cases are not described in detail so as not to obscure the presentation of the exemplary embodiments. In other cases, some process steps or operations known in the art may not be described at all. It should be understood that the following description focuses on characteristic features or elements according to various exemplary embodiments.

[0013] Anaphora is a word that refers to or replaces a word used earlier in a sentence. For example, in the sentence "I want this", the pronoun "this" is an anaphora that refers to an object. Techniques for analyzing and determining the object to which such a pronoun refers are known as anaphoric resolution. While anaphora is commonly used and understood in natural languages, the growing popularity of chatbots, virtual assistants, and other human-computer interfaces has created a need for techniques that can quickly and effectively resolve anaphora in computer-based systems, especially when user input data is limited.

[0014] The present invention proposes a technique to improve the accuracy of anaphoric resolution by using not only the content of the spoken text but also contextual data such as the user's position and orientation, the user's gaze, object metadata, etc. to specify the object that corresponds to the conversation target. The user position / orientation and gaze can be inferred from sensors such as a camera or a smart phone, while the object metadata can be stored in a local or remote database and can include, for example, the location, color, shape, etc. of the object that may be the object referenced in the anaphoric expression. Based on the collected contextual information, the system can then determine the most likely object to which the anaphoric expression refers, as described in more detail herein.

[0015] The present invention improves upon existing anaphora resolution techniques by incorporating contextual information about the user and the environment into natural language and data analytics. Highlights of the exemplary embodiments include improved anaphora resolution accuracy, thereby improving the accuracy and efficiency of natural language processing and text mining techniques. Use cases of the exemplary embodiments include anaphora resolution in natural language across a variety of settings, such as use in personal assistants, retail settings, work settings, presentation settings, and the like. A detailed description of the exemplary embodiments follows.

[0016] FIG. 1 illustrates an anaphora resolution system 100, according to an exemplary embodiment. According to an exemplary embodiment, the anaphora resolution system 100 may include one or more sensors 110, an object database server 120, and an anaphora resolution server 130, all of which may be interconnected via a network 108. While the programming and data of the exemplary embodiment may be stored and accessed remotely across several servers via the network 108, the programming and data of the exemplary embodiment may alternatively or additionally be stored locally on as little as one physical computing device, or among other computing devices not described. For example, in an embodiment, the anaphora resolver 132 and necessary components may be stored on the sensor 110 for local use without the need to connect to the network 108. The operation of the anaphora resolution system 100 is described in more detail herein.

[0017] In an exemplary embodiment, the network 108 may be a communication channel over which data can be transferred between connected devices. In an exemplary embodiment, the network 108 may be the Internet, representing a worldwide collection of networks and gateways for supporting communication between devices connected to the Internet. Additionally, the network 108 may utilize various types of connections, such as wired, wireless, fiber optic, etc., implemented as an intranet network, a local area network (LAN), a wide area network (WAN), or combinations thereof. In further embodiments, the network 108 may be a Bluetooth network, a Wi-Fi network, or combinations thereof. The network 108 may operate at frequencies including 2.4gHz and 5gHz Internet, short range wireless communications, Z-Wave, Zigbee, etc. In further embodiments, the network 108 may be a telecommunications network used to facilitate telephone calls between two or more parties comprising a land line network, a wireless network, a closed network, a satellite network, or combinations thereof. In general, the network 108 may represent any combination of connections and protocols that support communication between connected devices.

[0018] In an exemplary embodiment, the sensor 110 may be one or more devices capable of collecting data. In particular, the sensor 110 may be configured to collect contextual information within an environment in which the anaphora resolution system 100 is implemented, for example as shown in FIG. 3. In an embodiment, the sensor 110 may collect user data such as a user's position / orientation, movement, gaze, speaking motion, gesturing motion, and the like. In some embodiments, the sensor 110 may further collect object data such as object position, color, shape, and the like, and may be further configured to periodically update object metadata in the object database server 120. According to some embodiments, the sensor 110 is located within the environment, e.g., a camera, microphone, network, and the like, while in other embodiments, the sensor 110 may be mobile, such as a computer, a smart phone, a smart watch, smart glasses, smart contact lenses, and the like. In general, the sensor 110 may be any device capable of collecting user and / or object information within the environment. The sensor 110 is described in more detail with respect to FIGS. 2-5.

[0019] In an exemplary embodiment, the object database server 120 includes an object database 122 and may be an enterprise server, a laptop computer, a notebook, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a server, a personal digital assistant (PDA), a rotary phone, a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, an IoT device, or any other electronic device or computing system capable of receiving and sending data to and from other computing devices. Although the object database server 120 is shown as a single device, in other embodiments, the object database server 120 may be comprised of a cluster or multiple computing devices operating together or independently, such as in a modular manner. The object database server 120 is described in detail as a hardware implementation with reference to FIG. 5, as part of a cloud implementation with reference to FIG. 6, or utilizing a functional abstraction layer for processing with reference to FIG. 7, or a combination thereof.

[0020] The object database 122 may be a repository of information and metadata about one or more objects. In an embodiment, the one or more objects may represent the object to which the anaphora refers and may be real-world objects, such as people, places, things, as well as virtual objects such as those depicted on a screen, monitor, virtual / augmented / reduced reality, etc. Additionally, the object database 122 may include information and metadata describing one or more objects, such as the object's shape, color, size, barcode, stock keeping unit (SKU), Media Access Control (MAC) address, Internet Protocol (IP) address, location (e.g., coordinates within a 3D representation of the environment), textual description of the object, etc. In an embodiment, the object database 122 may be populated and periodically updated, for example, via user input or via reference to the sensor 110. The object database 122 is described in more detail with respect to FIGS. 2-5.

[0021] In an exemplary embodiment, the anaphora resolution server 130 may include an anaphora resolver 132 and may be an enterprise server, a laptop computer, a notebook, a tablet computer, a netbook computer, a PC, a desktop computer, a server, a PDA, a rotary phone, a touch-tone phone, a smart phone, a mobile phone, a virtual device, a thin client, an IoT device, or any other electronic device or computing system capable of receiving and transmitting data to and from other computing devices. Although the anaphora resolution server 130 is shown as a single device, in other embodiments, the anaphora resolution server 130 may be comprised of a cluster or multiple computing devices operating together or independently. The anaphora resolution server 130 is described in detail with reference to FIG. 5 as a hardware implementation, with reference to FIG. 6 as part of a cloud implementation, or with reference to FIG. 7 utilizing functional abstraction layers for processing, or a combination thereof.

[0022] In an exemplary embodiment, the anaphora resolver 132 may be a software and / or hardware program capable of extracting personal context data from the interactions and / or actions of one or more humans in an environment. In addition, the anaphora resolver 132 may determine whether the personal context data includes an anaphoric expression, and if so, may extract the anaphoric context data from the user interactions and / or actions. Furthermore, the anaphora resolver 132 may match the anaphoric expression to an object based on the extracted personal context data and the extracted anaphoric context data. The anaphora resolver 132 may further score and rank matches between the anaphoric expression and one or more objects, and further tune a model for scoring the matches. The anaphora resolver 132 is described in more detail with reference to FIGS. 2-5.

[0023] FIG. 2 illustrates an exemplary flowchart 200 illustrating the operation of the anaphora resolver 132 of the anaphora resolution system 100, according to an exemplary embodiment.

[0024] The anaphora resolver 132 may extract personal context data (step 202). In an exemplary embodiment, the anaphora resolver 132 may extract personal context data from one or more human interactions and / or actions in an environment to facilitate identification of the object of an anaphoric expression occurring between individuals. The personal context data may include data regarding the number of people in the environment, the identities of the individuals, the individual identified as speaking, the location of the person (e.g., coordinates in three-dimensional space), groups of people, the orientation of the person relative to the speaker / one or more objects, line of sight, etc. In an exemplary embodiment, the anaphora resolver 132 may detect and identify people using the sensors 110 in combination with techniques such as image and video analysis, audio analysis, template matching, face print or voice print identification, entity extraction, etc. In another embodiment, the sensors 110 are implemented as a network, such as a WiFi network, and individuals may be identified and locations estimated based on WiFi connectivity. Further, the anaphora resolver 132 can use techniques such as audio-to-video matching, triangulation, trilateration, spatial analysis, and temporal analysis to identify what the person is talking about, as well as the location, group, and orientation. In addition, the anaphora resolver 132 can reference sensors 110 implemented as smart devices or wearables such as smart glasses or a smart watch, to extract the person's location, gaze, movement, and the like. For example, the anaphora resolver 132 can determine gaze based on a video or image feed from a pair of smart glasses worn by the person, or determine a location in an area based on a smart watch.

[0025] Additionally, the individual context data may further include natural language data and / or utterances of one or more individuals detected within the environment, such as audio, text, and gesture interactions (following step 202). The anaphora resolver 132 may extract the natural language data by first detecting text, audio, or gestures emitted by the individual via the sensor 110, which may be in the form of a microphone, camera, smart device, or other sensor, mounted within the environment, mobile, worn by the user, etc. In embodiments where the natural language is audible and / or speech, the anaphora resolver 132 may implement the sensor 110 as a microphone and extract, translate, transcribe, etc. the detected audio. Additionally, the anaphora resolver 132 may implement the sensor 110 as a camera and utilize lip reading technology to extract the natural language. Similarly, the anaphora resolver 132 may be configured to implement the sensor 110 as a camera and extract natural language from gestures, such as pointing, waving, sign language, etc., by referencing a database that associates gestures and sign language with natural language. In an embodiment where the natural language is text, the anaphora resolver 132 may implement the sensor 110 as a smart device, e.g., a smart phone or laptop, and eavesdrop on text dialogue shared via messages, social networks, etc. In general, the anaphora resolver 132 may implement any technique and strategy for extracting personal context data via the sensor 110. After extracting audio, video, etc. of the individual's natural language, the anaphora resolver 132 may apply natural language processing and understanding techniques to the extracted natural language to determine the natural language conveyed.

[0026] To better illustrate the operation of the anaphora resolver 132, reference is made to the illustrative example shown in Figures 3-4 in which an individual 316 and an individual 318 are discussing items displayed on a shelf 202 in a retail setting. Here, the anaphora resolver 132 utilizes the sensor 110 as a combination camera and microphone device to extract a dialogue between the individuals in which the individual responds with "The black and round one" to the utterance "Which one would you like?". In addition, the anaphora resolver 132 uses the sensor 110 as a camera to extract the identity, location, and gaze of the individuals 316, 318 relative to the environment.

[0027] The anaphora resolver 132 can determine whether the individual's language includes one or more anaphoric expressions (determine 204). In an exemplary embodiment, the anaphora resolver 132 can determine whether the user's language includes one or more anaphoric expressions by utilizing a number of natural language processing techniques. In an embodiment, the anaphora resolver 132 can identify anaphoric expressions using a rule-based approach or, for example, templates and / or pattern matching. In addition, the anaphora resolver 132 can further implement machine learning techniques to learn correspondences between anaphors and other parts of speech, for example, using part-of-speech analysis, entity extraction, models, references to corpora, etc. However, in other embodiments, the anaphora resolver 132 can implement other techniques to detect anaphors within user dialogue or idioms.

[0028] Continuing with the illustrative example shown in FIGS. 3-4, in which the dialogue includes “Which one would you like?” and “The black and round one,” the anaphora resolver 132 determines, based on machine learning and template matching techniques, that the term “one” in both sentences is an anaphoric expression.

[0029] The anaphor resolver 132 can extract anaphoric context data from the personal context data (step 206). In an exemplary embodiment, the anaphoric context data includes items related to the identified anaphoric expression and the corresponding object. Such anaphoric context data can include the color, shape, location, size, time, etc. of the object to which the anaphor refers. To extract the anaphoric context data, the anaphoric resolver 132 can extract keywords and topics from the personal context data, such as audio or text that references color, size, shape, location, time, etc. In addition, the anaphoric resolver 132 can further extract anaphoric context data from other personal context data, such as where one or more people are pointing, the person's orientation, the person's gaze, etc. The anaphoric resolver 132 can extract anaphoric context data from one or more words, sentences, paragraphs, etc. adjacent or proximate to the identified anaphor. For example, the anaphoric resolver 132 can perform natural language processing using dependency analysis on n sentences before and after the identified anaphoric expression. Additionally, the anaphora resolver 132 records and associates such personal and anaphoric context data with the identified person, such that the anaphora resolver 132 can generate a personal profile to track the individual's interests, hobbies, tendencies, patterns, etc. The anaphora resolver 132 can further utilize the personal profile information in extracting anaphoric context data based on tendencies, probabilities, etc. by incorporating the profile data into future anaphora resolutions. For example, if an individual has frequently referred to watches in the past, there is a higher probability that the individual is currently referring to watches.

[0030] With reference to the illustrative example above involving the natural language of "Which one would you like?" and "The black and round one," the anaphora resolver 132 would extract anaphoric context data of "black" and "round" based on black and round being part of the color and shape of the topic, respectively. In addition, if the individual points or gestures towards a particular object, the anaphora resolver 132 would further extract information regarding the direction of the gesture made by the individual.

[0031] The anaphor resolver 132 may match the anaphoric expression to one or more objects (step 208). In an embodiment, the anaphor resolver 132 may match the anaphoric expression to an object by comparing both the extracted personal context data and the extracted anaphoric context data to the object data detailed by the object database 122. In an embodiment, the anaphor resolver 132 may compare the personal context data, such as position, orientation, line of sight, etc., to the location of the object detailed by the object database 122. For example, the anaphor resolver 132 may utilize the location and orientation of one or more persons to determine that the person is looking at a particular shelf and line of sight direction to determine that the person is looking at a display item at the top of the shelf. Additionally, the anaphor resolver 132 may compare the anaphoric context data, such as the color, shape, size, etc., of the object to the color, shape, size, etc., of the object data detailed by the object database 122. For example, the anaphora resolver 132 may compare the color, shape, size, or publication date described by the individual to the color, shape, size, or publication date of the object detailed by the object database 122. For each object considered in the comparison, the anaphora resolver 132 may determine an object match score for each comparison made (e.g., a score for each of the location, orientation, line of sight, color, shape, size, etc.) and may combine the object match scores to determine an overall object match score for each of the objects as they relate to each of the identified anaphoric expressions.

[0032] In an embodiment, the anaphor resolver 132 may consider any or all of the above object match scores, as well as any others, when determining an overall object match score for each object considered (following step 208). In an embodiment, the anaphor resolver 132 may consider the object match scores as features in a model where each feature is weighted when calculating the overall object match score. For example, features determined to highly correlate the anaphoric expression to the object may be weighted more heavily than features that do not. Then, for each object, the anaphor resolver 132 may, for example, multiply the features by the associated weights and sum the object match scores to determine an overall object match score. Such models may be generated and trained using techniques such as machine learning, neural nets, regression, etc. Where applicable, the anaphor resolver 132 may be configured to identify one or more objects to which the anaphor refers. Additionally, the anaphor resolver 132 may be configured to consider other anaphors when identifying the object of a particular anaphor. For example, if an individual says, "It's the one above that one," the anaphora resolver 132 may first determine what object "above that one" refers to before determining what "the one" is.

[0033] Referring again to the example introduced previously, the anaphora resolver 132 determines object match scores between the anaphoric expressions and shelf 302, clock 308, clock 310, cake 312, and lollipop 314, as detailed by object database 122. The anaphora resolver 132 further sums the object match scores to determine an overall object match score as a 95% match with clock 310, a 65% match with clock 308, a 30% match with lollipop 314, and a 10% match with cake 312.

[0034] The anaphor resolver 132 may score and rank the anaphoric object matches (step 210). Based on a comparison of the anaphoric expressions to the objects detailed by the object database 122, the anaphor resolver 132 may score the potential objects based on the probability that they are the object to which the anaphor refers. In addition, the anaphor resolver 132 may rank the scored potential objects and, in embodiments, associate the score, the personal context data, and the anaphoric context data with the object in the object database 122. In embodiments, the anaphor resolver 132 ranks the best matching objects as the most likely objects to which the anaphor refers. The anaphor resolver 132 may output the highest ranked objects to a user interface, another computing device, or the like. For example, the anaphor resolver 132 may ask the person whether it is the object to which it refers by text, push notification, email, or the like, or may use / transfer the object for use in natural language processing and understanding applications.

[0035] Referring to the example provided above, the anaphora resolver 132 ranks the clock 310 as the most likely object to which the anaphora "the black and round one" refers and utilizes the clock 310 in the natural language processing of the dialogue between the individuals 316, 318.

[0036] The anaphora resolver 132 may adjust the model (step 212). In an embodiment, the anaphora resolver 132 may adjust the model based on determining whether the correct object to which the anaphora refers was identified. For example, the anaphora resolver 132 may continue to extract the personal context data and the anaphora context data to reanalyze whether the correct object was originally identified. If the anaphora resolver 132 later determines that a different object is more likely to be the object referenced by the anaphora than the original object, the anaphora resolver 132 may adjust the weights associated with the features originally relied upon. In addition, the anaphora resolver 132 may reference the sensor 110 to determine whether the person ultimately selected, picked up, used, etc., the originally identified object, thereby confirming the identified object. In other embodiments, the anaphora resolver 132 may implement other methods of feedback loops, such as receiving user input by a human or administrator.

[0037] Referring again to the example introduced previously, the anaphora resolver 132 adjusts the weights associated with the features based on the person ultimately selecting clock 308 over clock 310.

[0038] FIG. 3 illustrates an exemplary schematic diagram of an environment 300 in which the anaphora resolution system 100 is implemented, according to an exemplary embodiment.

[0039] 3 illustrates an embodiment in which the anaphora resolution system 100 is implemented within a retail setting. The exemplary environment 300 may include a retail shelf 302, a user 316, a user 318, and a sensor 110. The retail shelf 302 may include a middle shelf plate 304 and a bottom shelf plate 306 for displaying objects such as merchandise. The middle shelf plate 304 may include a gray clock 308 and a black clock 310, while the bottom shelf plate 306 may include a cake 312 and a lollipop 314. In the exemplary environment 300, the sensor 110 may be configured to detect and measure a user's natural language and gesture data such as the user's voice, text, or actions. In addition, the sensor 110 may be configured to detect and measure personal contextual data such as position, motion, gaze, movement, gestures, etc., for the user 316 and the user 318 within the environment 300. In some embodiments, the sensor 110 may be further configured to detect and measure object contextual data regarding the objects 308, 310, 312, and 314, such as object position, shape, size, color, movement, etc.

[0040] 4 illustrates an exemplary table 400 representing data stored in object database server 120 of anaphora resolution system 100, according to an exemplary embodiment. More specifically, table 400 illustrates an exemplary products table detailed by object database 122 that corresponds to objects in environment 300 of FIG.

[0041] FIG 5 illustrates a block diagram of devices in the anaphora resolution system 100 of FIG 1 in accordance with an illustrative embodiment. It should be understood that FIG 5 is only intended to provide an illustration of one implementation and is not intended to imply limitation with respect to the environment in which different embodiments may be implemented. Many variations may be made to the depicted environment.

[0042] As used herein, a device may include one or more processors 02, one or more computer readable RAMs 04, one or more computer readable ROMs 06, one or more computer readable storage media 08, device drivers 12, read / write drives or interfaces 14, and network adapters or interfaces 16, all interconnected through a communications fabric 18. The communications fabric 18 may be implemented in any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and other hardware components in the system.

[0043] One or more operating systems 10, and one or more application programs 11 are stored in one or more of the computer-readable storage media 08 for execution by one or more of the processors 02 via one or more of the respective RAMs 04 (which typically include cache memory). In the illustrated embodiment, each of the computer-readable storage media 08 may be a magnetic disk storage device of an internal hard drive, a CD-ROM, a DVD, a memory stick, a magnetic tape, a magnetic disk, an optical disk, a semiconductor storage device such as RAM, ROM, EPROM, flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0044] A device as used herein may also include a R / W drive or interface 14 for reading from and writing to one or more portable computer readable storage media 26. Application programs 11 on the device may be stored on one or more of the portable computer readable storage media 26 and read via the respective R / W drive or interface 14 and loaded into the respective computer readable storage medium 08.

[0045] The devices used herein may also include a network adapter or interface 16, such as a TCP / IP adapter card or a wireless communication adapter (such as a 4G wireless communication adapter using OFDMA technology). The application programs 11 on the computing device may be downloaded to the computing device from an external computer or external storage device via a network (e.g., the Internet, a local area network, or other wide area network or wireless network) and the network adapter or interface 16. From the network adapter or interface 16, the programs may be loaded into the computer-readable storage medium 08. The network may comprise copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof.

[0046] The devices used herein may also include a display screen 20, a keyboard or keypad 22, and a computer mouse or touchpad 24. The device driver 12 interfaces to the display screen 20 for imaging, to the keyboard or keypad 22, to the computer mouse or touchpad 24, or to the display screen 20 for alphanumeric input and pressure sensing of user selection, or a combination thereof. The device driver 12, the R / W drive or interface 14, and the network adapter or interface 16 may comprise hardware and software (stored in the computer readable storage medium 08 or ROM 06, or both).

[0047] The programs described herein are identified based on the application for which they are implemented in particular ones of the exemplary embodiments. However, it should be understood that the specific program nomenclature herein is used merely for convenience, and thus the exemplary embodiments should not be limited to use only in any particular application identified and / or suggested by such nomenclature.

[0048] Based on the above, a computer system, method, and computer program product have been disclosed. However, numerous modifications and substitutions can be made without departing from the scope of the exemplary embodiments. Thus, the exemplary embodiments are disclosed by way of example and not by way of limitation.

[0049] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, exemplary embodiments may be implemented in conjunction with any other type of computing environment now known or later developed.

[0050] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0051] The characteristics are as follows: On-demand self-service. Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without requiring human interaction with the provider of the service. Broad network access. Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (e.g., mobile phones, laptops, and PDAs). Resource Pooling. A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers typically have no control or knowledge over the exact location of the provided resources, although there is a sense of location independence in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity. Capabilities are rapidly and elastically provisioned, sometimes automatically, and can be rapidly scaled out, rapidly released, and rapidly scaled in. To the consumer, the capabilities available for provisioning often appear unlimited, and can be purchased in any quantity at any time. Metering Services. Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (storage, processing, bandwidth, active user accounts, etc.). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0052] The service model is as follows: Software as a Service (SaaS). The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even personal application capabilities, except for limited user-specific application configuration settings. Platform as a Service (PaaS). The functionality offered to the consumer is the deployment onto a cloud infrastructure of applications that the consumer has created or acquired, written using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but has control over the deployed applications and, potentially, the configuration of the application hosting environment. Infrastructure as a Service (IaaS). The capability provided to the consumer is to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases, limited control over select networking components (e.g., host firewalls).

[0053] The deployment model is as follows: Private Cloud: Cloud infrastructure is operated solely for the organization, is managed by the organization or a third party, and may reside on-premises or off-premises. Community Cloud. Cloud infrastructure is shared across multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, compliance considerations). It may be managed by the organization or a third party and may reside on or off premises. Public Cloud. Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud. A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that allow data and application portability (e.g., cloud bursting for load balancing between clouds).

[0054] A cloud computing environment is a service oriented approach that focuses on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0055] Referring now to FIG. 6, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 40 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, can communicate. The nodes 40 can communicate with each other. The nodes can be physically or virtually grouped (not shown) into one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on the local computing device. The types of computing devices 54A-54N shown in FIG. 6 are intended as examples only, and it will be understood that the computing nodes 40 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network addressable connections (e.g., using a web browser).

[0056] Referring now to Figure 7, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 6) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 7 are intended to be illustrative only, and example embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0057] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, RISC (Reduced Instruction Set Computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0058] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0059] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and billing 82 tracks costs as resources are utilized within the cloud computing environment and charges or bills for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, and protection of data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance provisioning and procurement of cloud computing resources for anticipated future requirements according to SLAs.

[0060] The workload layer 90 provides examples of functions for which cloud computing environments are utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and anaphora processing 96.

[0061] Exemplary embodiments may be systems, methods, and / or computer program products at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0062] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as raised configurations in grooves with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0063] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0064] The computer readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuitry, object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0065] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0066] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, capable of directing a computer, programmable data processing apparatus or other device, or combination thereof, to function in a particular manner, such that a computer readable storage medium having stored thereon comprises an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0067] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0068] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of an instruction, which comprises one or more executable instructions for performing a specified logical function. In some alternative implementations, the functions shown in the blocks may differ from the order shown in the figures. For example, two blocks shown in succession may in fact be performed as one step, executed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

Claims

1. 1. A computer-implemented method for resolving anaphora, the method comprising: Extracting, from text, audio, or gestures uttered by an individual, which is an individual expression, interaction data or action data of one or more individuals within an environment, which is individual context data; determining whether the personal usage includes anaphoric expressions by applying natural language processing to the personal context data to determine whether the personal usage includes anaphoric expressions; extracting anaphoric context data including keywords and topics from the personal context data based on determining that the personal usage includes the anaphoric expression; and identifying an object of the one or more objects to which the anaphoric expression refers based on a comparison of the personal context data and the anaphoric context data with object metadata detailing the one or more objects.

2. Identifying the object of the one or more objects to which the anaphoric expression refers includes: extracting object metadata corresponding to the one or more objects; The method of claim 1 , further comprising: comparing the object metadata corresponding to the one or more objects to the personal context data and the anaphoric context data.

3. determining an object match score for each comparison of the object metadata to the personal context data and the anaphoric context data; and determining an overall object match score for each of the one or more objects as the object match score is relevant to resolving the anaphoric expression.

4. Determining the overall object match score for each of the one or more objects as the object match score is relevant to resolving the anaphoric expression, further comprising: generating a model that weights each comparison; multiplying said weight by the object match score of each said comparison; and summing the multiplied and weighted values.

5. The method of claim 1 , wherein the anaphoric context data includes data selected from the group consisting of object color, object shape, object placement, object size, and object duration.

6. 2. The method of claim 1, wherein the individual context data includes data selected from the group consisting of a number of individuals in an environment, identities of the individuals, individuals identified as speaking, relative positions of the individuals, groups of the individuals, orientation of the individuals relative to a speaker, orientation of the individuals relative to one or more objects, and line of sight of the individuals.

7. generating a personal profile for at least one individual generating the personal usage; The method of claim 6 , wherein the personal contextual data comprises the personal profile.

8. A computer program product for causing a computer to execute the method according to any one of claims 1 to 7.

9. A computer readable storage medium having the computer program according to claim 8 stored therein.

10. A computer system that executes the method according to any one of claims 1 to 7 by means of computer hardware.

Images