Identifying entity-attribute relationships

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, facilitate identifying entity-attribute relationships in a text corpus. The method includes determining whether an attribute in a candidate entity-attribute pair is an actual attribute of an entity in the entity-attribute candidate pair. This may include generating embeddings for words in a sentence including the entity and the attribute and generating. This may also include generating an attribute distribution embedding for the entity based on other attributes associated with the entity, and generating an attribute distribution embedding for the attribute based on known attributes associated with known entities of the attribute. Based on these embeddings, a feed-forward network may determine whether an attribute in an entity-attribute candidate pair is an actual attribute of an entity in the entity-attribute candidate pair.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is an international application and claims the benefit of U.S. Application No. 16 / 504,068, filed on July 5, 2019. The disclosure of the foregoing application is incorporated herein by reference in its entirety. Background Art

[0003] This specification relates to identifying entity-attribute relationships in a text corpus.

[0004] Search-based applications (e.g., search engines, knowledge bases) are designed to identify resources (e.g., web pages, images, text documents, and multimedia content) that are relevant to a user's information needs and present information about the resources in a manner that is most useful to the user. One way in which a search-based application can present information about an identified resource is in the form of structured search results. Structured search results typically present a list of attributes with answers for an entity specified in a user request (e.g., a query). For example, in response to a query for "Kevin Durant," structured search results may include attributes about "Kevin Durant" (such as salary, team, year of birth, family, etc.), as well as answers that provide information about those attributes.

[0005] Building such structured search results often requires identifying entity-attribute relationships. Entity-attribute relationships are a special case of textual relationships between pairs of terms. The first term in the pair is the entity, which can be a person, place, organization, concept, etc. The second term in the pair is the attribute, which is a string that describes some aspect or characteristic of the entity. Examples of attributes can include the "date of birth" of a person, the "population" of a country, the "salary" of an athlete, or the "CEO" of an organization.

[0006] However, large amounts of data must often be processed in order to identify suitable search results. This can use a large amount of processing power. In addition, the need to process such large amounts of data can result in slow results, or compromises in processing that result in poor quality results. Summary of the invention

[0007] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method, the method comprising the following operations: obtaining an entity-attribute candidate pair defining an entity and an attribute, wherein the attribute is a candidate attribute of the entity; determining whether the attribute is an actual attribute of the entity in the entity-attribute candidate pair based on a set of sentences including the entity and the attribute, the determination comprising: generating embeddings for words in the set of sentences including the entity and the attribute; generating an attribute distribution embedding for the entity using known entity-attribute pairs, wherein the attribute distribution embedding for the entity specifies the embedding for the entity based on other attributes associated with the entity from the known entity-attribute pairs; generating an attribute distribution embedding for the attribute using known entity-attribute pairs, wherein the attribute distribution embedding for the attribute specifies the embedding for the attribute based on a known attribute associated with a known entity in the known entity-attribute pair; determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair based on the embeddings for words in the set of sentences, the attribute distribution embedding for the entity, and the attribute distribution embedding for the attribute. Other embodiments of this aspect include corresponding systems, devices, apparatuses, and computer programs configured to perform the actions of the method. The computer program (eg, instructions) may be encoded on a computer storage device.These and other embodiments may each optionally include one or more of the following features.

[0008] In some embodiments, generating embeddings for words in a set of sentences including entities and attributes includes: generating a first vector representation of a first embedding specifying the words between the entities and the attributes in the set of sentences; generating a second vector representation specifying a second embedding for the entities based on the set of sentences; and generating a third vector representation specifying a third embedding for the attributes based on the set of sentences.

[0009] In some implementations, generating an attribute distribution embedding for an entity using known entity-attribute pairs includes: generating a fourth vector representation specifying an attribute distribution embedding for the entity using the known entity-attribute pairs.

[0010] In some implementations, generating an attribute distribution embedding for the attribute using the known entity-attribute pairs includes: generating a fifth vector representation specifying the attribute distribution embedding for the attribute using the known entity-attribute pairs.

[0011] In some embodiments, determining whether an attribute in an entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair based on embeddings for words in a sentence set, attribute distribution embeddings for entities, and attribute distribution embeddings for attributes includes: determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair based on a first vector representation, a second vector representation, a third vector representation, a fourth vector representation, and a fifth vector representation.

[0012] In some embodiments, a feedforward network is used to perform: determining whether an attribute in an entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair based on the first vector representation, the second vector representation, the third vector representation, the fourth vector representation, and the fifth vector representation.

[0013] In some embodiments, determining whether an attribute in an entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair based on the first vector representation, the second vector representation, the third vector representation, the fourth vector representation, and the fifth vector representation includes: generating a single vector representation by cascading the first vector representation, the second vector representation, the third vector representation, the fourth vector representation, and the fifth vector representation; inputting the single vector representation into a feedforward network; and determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair through the feedforward network and using the single vector representation.

[0014] In some embodiments, generating a fourth vector representation specifying an attribute distribution embedding for an entity using known entity-attribute pairs includes: identifying a set of attributes associated with the entity in the known entity-attribute pairs, where the attribute set does not include the attribute; and generating an attribute distribution embedding for the entity by calculating a weighted sum of the attributes in the attribute set.

[0015] In some embodiments, using known entity-attribute pairs, generating a fifth vector representation that specifies an attribute distribution embedding for an attribute includes: identifying a set of entities from the known entity-attribute pairs using the attribute; for each entity in the entity set, identifying a set of attributes associated with the entity, wherein the attribute set does not include the attribute; and generating an attribute distribution embedding for the attribute by calculating a weighted sum of the attributes in the attribute set.

[0016] Compared to the model-based entity-attribute recognition techniques of the prior art, specific embodiments of the subject matter described in this specification can be implemented to achieve the advantage of identifying more accurate entity-attribute relationships. The entity-attribute recognition techniques of the prior art use various model-based methods (e.g., natural language processing (NLP) features, distant supervision, and traditional machine learning models) that identify entity-attribute relationships by representing entities and attributes based on the data (e.g., sentences) in which these terms appear. In contrast, the innovation described in this specification not only uses information about how entities and attributes are described in the data in which these terms appear, but also uses other attributes known to be associated with these terms to represent entities and attributes to identify entity-attribute relationships in a data set. This supports representing entities and attributes with attributes shared by similar entities, thereby improving the accuracy of identifying entity-attribute relationships, which otherwise cannot be discerned by only considering the sentences in which these terms appear. Therefore, when a user performs a search, more useful information can be obtained. In particular, a wider range of search terms will cause more relevant information to be identified. Therefore, repeated searches are not necessarily required. Therefore, to obtain the desired search results, less processing may be required.

[0017] For example, consider a scenario where a data set includes: a sentence with two entities "Ronaldo" and "Messi" described using the "record" attribute; and a sentence where the entity "Messi" is described using the "goals" attribute. In this scenario, the prior art can identify the following entity-attribute pairs: (Ronaldo, record), (Messi, record), and (Messi, goals). The innovations described in this specification go beyond these prior art methods by identifying entity-attribute relationships that may not be easily discernible by how the terms are used in the data set. Using the above example, the innovations described in this specification determine that "Ronaldo" and "Messi" are similar entities because they share the "record" attribute, and then use the "goals" attribute to represent the "record" attribute. In this way, innovations such as those described herein may enable identification of entity-attribute relationships, such as (Cristiano, Goals), even though such relationships may not be readily discernible from a data set.

[0018] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an example environment for extracting entity-attribute relationships.

[0020] Figure 2 is a flow chart of an example process for identifying entity-attribute relationships.

[0021] Figure 3 is a block diagram of an example computer system.

[0022] Like reference numerals and designations in the various drawings refer to like elements. DETAILED DESCRIPTION

[0023] This specification relates to identifying entity-attribute relationships in a text corpus.

[0024] As further described in this specification, candidate entity-attribute pairs (wherein the attributes are candidate attributes of the entity) are input to a classification model. The classification model, which includes a path embedding engine, a distribution representation engine, an attribute distribution engine, and a feedforward network, determines whether the attributes in the candidate entity-attribute pair are actual attributes of the entity in the candidate entity-attribute pair.

[0025] The path embedding engine generates a vector that represents the embedding of the path or word that connects the joint occurrence of entities and attributes in a set of sentences (e.g., 30 or more sentences) in the dataset. The distribution representation engine generates a vector representing the embedding for entity terms and attribute terms based on the context in which these terms appear in the sentence set. The attribute distribution engine generates a vector representing the embedding for the entity and another vector representing the embedding for the attribute. The attribute distribution engine embeds the entity based on other attributes known to be associated with the entity in the dataset (i.e., attributes other than the candidate attributes). The attribute distribution engine embeds the attribute based on other attributes of the candidate attribute associated with the known entity.

[0026] The classification model concatenates the vector representations from the path embedding engine, the distribution representation engine, and the attribute distribution engine into a single vector representation. Since a single vector representation is used for these concatenated vectors, less data needs to be stored. The classification model then inputs the single vector representation into a feedforward network, which uses the single vector representation to determine whether the attributes in the candidate entity-attribute pair are actual attributes of the entity in the candidate entity-attribute pair. If the feedforward network determines that the attributes in the candidate entity-attribute pair are actual attributes of the entity in the candidate entity-attribute pair, the candidate entity-attribute pair is stored in the knowledge base along with other known / actual entity-attribute pairs.

[0027] Reference below Figures 1 to 3 These and additional features are described in more detail.

[0028] Figure 1 1 is a block diagram of an example environment for extracting entity attribute relationships. Environment 100 includes a classification model 114 that determines, for candidate entity-attribute pairs in knowledge base 104, whether an attribute in a candidate entity-attribute pair is an actual attribute of an entity in the candidate pair. In some embodiments, classification model 114 is a neural network model, and its components / engines (and their corresponding operations) are described below. It should be appreciated that other types of supervised and / or unsupervised machine learning models may also be used to implement classification model 114.

[0029] The knowledge base 104, which may include one or more databases (or other appropriate data storage structures) stored in one or more non-transitory data storage media (e.g., hard drives, flash memories, etc.), stores a set of candidate entity-attribute pairs. The candidate entity-attribute pairs may be obtained using a set of content in the form of text documents (e.g., web pages, news articles, etc.) obtained from a data source 102, which may include any content source, such as a news website, a data aggregator platform, a social media platform, etc. In some embodiments, the data source 102 obtains the news article from the data aggregator platform. In some embodiments, the data source 102 may use a model (e.g., a supervised or unsupervised machine learning model, a natural language processing model) to generate a set of candidate entity-attribute pairs by extracting sentences from the article and tokenizing and labeling the extracted sentences (e.g., as entities and attributes) using part-of-speech tags and dependency parse tree tags. In some embodiments, the data source 102 may input the extracted sentences into a machine learning model, which may be trained, for example, using a set of training sentences and their associated entity-attribute pairs. This machine learning model can then output candidate entity-attribute pairs for the input extracted sentence.

[0030] The data source 102 stores the candidate entity-attribute pairs and the sentences including the words of the candidate entity-attribute pairs extracted by the data source 102 in the knowledge base 104. In some implementations, the candidate entity-attribute pairs are stored in the knowledge base 104 only when the number of sentences in which the entity and the attribute exist satisfies (e.g., reaches or exceeds) a threshold number of sentences (e.g., 30 sentences). Therefore, the amount of stored data is reduced by setting a threshold for determining whether to store data.

[0031] The classification model 114 determines whether an attribute in a candidate entity-attribute pair (stored in the knowledge base 104) is an actual attribute of the entity in the candidate entity-attribute pair. The classification model 114 includes a path embedding engine 106, a distribution representation engine 108, an attribute distribution engine 110, and a feedforward network 112. As used herein, the term engine refers to a data processing device that performs a set of tasks. Figure 2 The operation of each of these engines of the classification model 114 in determining whether an attribute in a candidate entity-attribute pair is an actual attribute of an entity is described.

[0032] Figure 2 2 is a flow chart of an example process 200 for identifying entity attribute relationships. The operation of process 200 is described below as follows: Figure 1 The operations of process 200 are performed by components of the system described and illustrated in the description. The operations of process 200 are described below for illustration purposes only. The operations of process 200 may be performed by any suitable device or system (e.g., any suitable data processing device). The operations of process 200 may also be implemented as instructions stored on a non-transitory computer-readable medium. Execution of the instructions causes one or more data processing devices to perform the operations of process 200.

[0033] The knowledge base 104 obtains entity-attribute candidate pairs from the data source 104, such as reference Figure 1 Described (at 202).

[0034] The knowledge base 104 obtains a sentence set including words of entities and attributes in the candidate entity-attribute pairs from the data source 102, as described in the above reference. Figure 1 as described (at 204).

[0035] The classification model 114 determines whether the candidate attribute is an actual attribute of the candidate entity based on the sentence set and the candidate entity-attribute pair. In some embodiments, the sentence set can be a large number of sentences, such as 30 or more sentences. The classification model makes this determination by performing the following operations: (1) generating embeddings for words in the sentence set including entities and attributes, which are described in more detail below with reference to operations 206, 208, and 210; (2) using known entity-attribute pairs, generating attribute distribution embeddings for entities, which are described in more detail below with reference to operation 212; (3) using known entity-attribute pairs, generating attribute distribution embeddings for attributes, which are described in more detail below with reference to operation 214; and (4) determining whether the attributes in the entity-attribute candidate pair are actual attributes of the entities in the entity-attribute candidate pair based on the embeddings for the words in the sentence set, the attribute distribution embeddings for the entities, and the attribute distribution embeddings for the attributes, which are described in more detail below with reference to operation 216. Operations 206 to 216 are described below.

[0036] The path embedding engine 106 generates a first vector representation of a first embedding of words between entities and attributes specified in the sentence set (at 206). The path embedding engine 106 detects relationships between candidate entity-attribute terms by embedding paths or words that connect joint occurrences of these terms in the sentence set. For example, for the phrase "snake is areptile", the path embedding engine 106 generates an embedding for the path "is a", which can then be used to detect, for example, genus-species relationships, which can then be used to identify other entity-attribute pairs. By generating such a path, less processing is required to perform analysis and detection of such terms.

[0037] The path embedding engine 106 performs the following operations to generate embeddings of words between entities and attributes in a set of sentences. For each sentence in the set of sentences, the path embedding engine 106 first extracts the dependency path (which specifies the set of words) between the entity and the attribute. The path embedding engine 106 converts the sentence from a string to a list, where the first term is the entity and the last term is the attribute (or alternatively, the first term is the attribute and the last term is the entity). Each term in the dependency path (which is also called an edge) is represented using the following features: the term's lemma, part-of-speech tag, dependency label, and the direction of the dependency path (left, right, or root). Each of these features is embedded and concatenated to produce a vector representation (V e ), which includes a vector sequence (V1, V pos 、V dep 、V dir ), as shown by the following equation:

[0038] .

[0039] The path embedding engine 106 then inputs the vector sequence of terms or edges in each path into a long short-term memory (LSTM) network, which produces a single vector representation (V s ), as shown by the following equation:

[0040]

[0041] This single vector representation reduces the amount of data that needs to be stored, processed, or transmitted because a single vector uses much less data than would be required for all the vectors in the single vector representation.

[0042] Finally, the path embedding engine 106 inputs a single vector representation for all sentences in the sentence set into the attention mechanism, which determines the sentence representation (V sents(e、a) ), as shown by the following equation:

[0043]

[0044] The distributed representation engine 108 generates a second vector representation for the entity and a third vector representation for the attribute based on the set of sentences (at 208 and 210). The distributed representation engine 108 detects the relationship between the candidate entity-attribute terms based on the attributes of the candidate entity-attribute pair and the context in which the entity occurs in the set of sentences. The vector-based approach for this processing reduces the amount of processing and therefore increases the speed of obtaining results. For example, the distributed representation engine 108 may determine that the entity "New York" is used in a set of sentences in a manner that indicates that the entity refers to a city or state in the United States. As another example, the distributed representation engine 108 may determine that the attribute "capital" is used in a set of sentences in a manner that indicates that the attribute refers to an important city in a state or country. Therefore, the distributed representation engine 108 uses the context in which the entity appears (i.e., the set of sentences) to generate a vector representation (V) that specifies the embedding for the entity. e Similarly, the distribution representation engine 108 uses the set of sentences in which the attribute appears to generate a vector representation (V ) specifying the embedding for the attribute. a ).

[0045] The attribute distribution engine 110 generates a fourth vector representation specifying an attribute distribution embedding for the entity using the known entity-attribute pairs (at 212). The known entity-attribute pairs stored in the knowledge base 104 are entity-attribute pairs for which each attribute in the entity-attribute pair has been confirmed (e.g., by the classification model 114 using previous processing or based on human evaluation) to be an actual attribute of the entity in the entity-attribute pair.

[0046] In some embodiments, the attribute distribution engine 110 performs the following operations to determine an attribute distribution embedding that specifies an embedding for an entity using some (e.g., the most common) or all of the other known attributes among the known entity-attribute pairs associated with the entity. For an entity in an entity-attribute candidate pair, the attribute distribution engine 110 identifies other attributes associated with the entity in the known entity-attribute pair (i.e., attributes other than the attributes included in the entity-attribute candidate pair). For example, for the entity "Michael Jordan" in the candidate entity-attribute pair (Michael Jordan, famous), the attribute distribution engine 110 can use known entity-attribute pairs about Michael Jordan, such as (Michael Jordan, wealthy) and (Michael Jordan, record), to identify attributes such as wealthy and record.

[0047] The attribute distribution engine 110 then generates an embedding for the entity by computing a weighted sum of the identified known attributes (as described in the previous paragraph), where the weights are learned using / through an attention mechanism, as shown in the following equation:

[0048]

[0049] The attribute distribution engine 110 generates a fifth vector representation specifying an attribute distribution embedding for the attribute using the known entity-attribute pairs (at 214). In some embodiments, the attribute distribution engine 110 performs operations to determine a representation for the attribute based on some (e.g., the most common) or all of the known attributes associated with the known entities of the candidate attribute. For an attribute in an entity-attribute candidate pair, the attribute distribution engine 110 identifies a known entity among the known entity-attribute pairs that has the attribute. For each identified known entity, the attribute distribution engine 110 identifies other attributes associated with the entity in the known entity-attribute pair (i.e., attributes other than the attributes included in the entity-attribute candidate pair). In some embodiments, the attribute distribution engine 110 may identify a subset of attributes from the identified attributes by: (1) ranking the attributes based on the number of known entities associated with each attribute (e.g., assigning a higher ranking to an attribute associated with a greater number of entities than an attribute associated with a smaller number of entities); and (2) then selecting a threshold number of attributes based on the ranking (e.g., selecting the top five attributes).

[0050] The attribute distribution engine 110 then generates an embedding for the attribute by computing a weighted sum of (all or a subset of) the identified known attributes (as described in the previous paragraph), where the weights are learned using / through an attention mechanism, as shown in the following equation:

[0051]

[0052] It should be appreciated that the embeddings generated by operations 206 through 214 are typically generated in parallel by the respective engines (106, 108, 110).

[0053] The feed-forward network 112 determines whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair based on the vector representation (at 216). In some embodiments, the feed-forward network 112 concatenates each of the vector representations output by the path embedding engine 106, the distribution representation engine 108, and the attribute distribution engine 110 into a single vector representation (V (e、a) ), as shown by the following equation:

[0054]

[0055] Using a single vector representation of the input, the feedforward network 112 outputs whether the attribute in the candidate entity-attribute pair is the actual attribute of the entity in the candidate entity-attribute pair. In some embodiments, the output of the feedforward network 112 can be binary. For example, when the attribute in the candidate entity-attribute pair is the actual attribute of the entity in the candidate entity-attribute pair, the feedforward network 112 can output "yes", and when the attribute in the candidate entity-attribute pair is not the actual attribute of the entity in the candidate entity-attribute pair, the feedforward network 112 can output "no". To produce such a binary output, the amount of processing required to obtain such a result is reduced due to the various simplifications made with the aid of the above process. In some embodiments, the output of the feedforward network 112 can be a confidence value (e.g., a value ranging from 0 to 1), where 0 specifies that the attribute in the candidate entity-attribute pair is not the actual attribute of the entity in the candidate entity-attribute pair, and 1 specifies that the attribute in the candidate entity-attribute pair is the actual attribute of the entity in the candidate entity-attribute pair.

[0056] In some embodiments, distant supervision is used to train the feedforward network 112. The training uses the classification model 114 to perform the above processing on candidate entity-attribute pairs that have been identified as true pairs (i.e., the attributes in their candidate entity-attribute pairs have been identified (e.g., based on human evaluation or previous processing by the feedforward network 112) as actual attributes of the entities in the candidate entity-attribute pairs).

[0057] If the output of the feed-forward network 112 specifies that an attribute in the entity-attribute candidate pair is an actual attribute (e.g., if the feed-forward network 112 outputs a "yes" indicator as described above) or has a high probability (e.g., as described above, the feed-forward network 112 outputs a confidence value that meets or exceeds a certain threshold, such as 0.8), the feed-forward network 112 stores the entity-attribute candidate pair as an actual entity-attribute pair in the knowledge base 104. This provides certainty about the data stored. In addition, it ensures that only relevant data is stored, which in turn improves the quality of the stored data and reduces the total amount of stored data.

[0058] Figure 3 300. The system 300 includes a processor 310, a memory 320, a storage device 330, and an input / output device 340. Each of the components 310, 320, 330, and 340 can be interconnected, for example, using a system bus 350. The processor 310 is capable of processing instructions executed within the system 300. In one embodiment, the processor 310 is a single-threaded processor. In another embodiment, the processor 310 is a multi-threaded processor. The processor 310 is capable of processing instructions stored in the memory 320 or on the storage device 330.

[0059] Memory 320 stores information within system 300. In one embodiment, memory 320 is a computer readable medium. In some embodiments, memory 320 is a volatile memory unit. In another embodiment, memory 320 is a non-volatile memory unit.

[0060] The storage device 330 can provide mass storage for the system 300. In some embodiments, the storage device 330 is a computer-readable medium. In various embodiments, the storage device 330 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) over a network, or some other mass storage device.

[0061] The input / output device 340 provides input / output operations for the system 300. In some embodiments, the input / output device 340 may include one or more network interface devices (e.g., Ethernet cards), serial communication devices (e.g., RS-232 ports), and / or wireless interface devices (e.g., 802.11 cards). In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices such as keyboards, printers, and display devices 360. However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, set-top TV client devices, etc.

[0062] Although already Figure 3 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented with other types of digital electronic circuits or with computer software, firmware or hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them.

[0063] The subject matter and the embodiments of the operations described in this specification may be implemented with digital electronic circuits or with computer software, firmware or hardware including the structures disclosed in this specification and their structural equivalents or with a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more modules of one or more computer programs, i.e., computer program instructions, encoded on a computer storage medium (or multiple media) for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices).

[0064] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0065] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip or a plurality of programmable processors, computers, systems on a chip or a combination of the foregoing. The apparatus may include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0066] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communication network.

[0067] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and the apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0068] As an example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential element of a computer is a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operationally coupled to receive data from one or more large-capacity storage devices (e.g., disks, magneto-optical disks or optical disks) for storing data, or transfer data to one or more large-capacity storage devices for storing data, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices including, by way of example: semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0069] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0070] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser that a user can use to interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0071] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user interacting with the client device). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server.

[0072] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or that may be claimed, but rather as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0073] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0074] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method comprising: obtaining an entity-attribute candidate pair defining an entity and an attribute, wherein the attribute is a candidate attribute of the entity; as well as Based on a sentence set including the entity and the attribute, determining whether the attribute is an actual attribute of the entity in the entity-attribute candidate pair, the determining comprising: generating embeddings for words in the set of sentences including the entity and the attribute, generating an attribute distribution embedding for the entity using known entity-attribute pairs, wherein the attribute distribution embedding for the entity specifies an embedding for the entity based on other attributes associated with the entity from the known entity-attribute pairs, generating an attribute distribution embedding for the attribute using the known entity-attribute pair, wherein the attribute distribution embedding for the attribute specifies an embedding for the attribute based on known attributes associated with a known entity for the attribute in the known entity-attribute pair, and Based on the embeddings for the words in the sentence set, the attribute distribution embeddings for the entities, and the attribute distribution embeddings for the attributes, it is determined whether the attributes in the entity-attribute candidate pairs are actual attributes of the entities in the entity-attribute candidate pairs.

2. The computer-implemented method of claim 1 , wherein generating the embeddings for the words in the set of sentences including the entity and the attribute comprises: generating a first vector representation specifying a first embedding of a word between the entity and the attribute in the set of sentences; generating, based on the set of sentences, a second vector representation specifying a second embedding for the entity; as well as Based on the set of sentences, a third vector representation specifying a third embedding for the attribute is generated.

3. The computer-implemented method of claim 2, wherein: Generating the attribute distribution embedding for the entity includes: generating a fourth vector representation using the known entity-attribute pairs, the fourth vector representation specifying the attribute distribution embedding for the entity; and Generating the attribute distribution embedding for the attribute includes generating a fifth vector representation using the known entity-attribute pairs, the fifth vector representation specifying the attribute distribution embedding for the attribute.

4. The computer-implemented method of claim 3, wherein: Determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair comprises: Based on the first vector representation, the second vector representation, the third vector representation, the fourth vector representation and the fifth vector representation, it is determined whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair.

5. The computer-implemented method of claim 4, wherein: The operation of determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair is performed using a feed-forward network.

6. The computer-implemented method of claim 5, wherein: Determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair comprises: combining the first vector representation, the second vector representation, the third vector representation, the fourth vector representation, and the fifth vector representation into a single vector representation; inputting the single vector representation into the feed-forward network; and By means of the feed-forward network and using the single vector representation, it is determined whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair.

7. A computer-implemented method according to any one of claims 3 to 6, wherein: Generating the fourth vector representation comprises: identifying a set of attributes associated with the entity in the known entity-attribute pair, wherein the set of attributes does not include the attribute; and The attribute distribution embedding for the entity is generated by calculating the weighted sum of the attributes in the attribute set.

8. The computer-implemented method of any one of claims 3 to 6, wherein: Generating the fifth vector representation comprises: Using the attributes, identifying a set of entities from the known entity-attribute pairs; For each entity in the set of entities, identifying a set of attributes associated with the entity, wherein the set of attributes does not include the attribute; and The attribute distribution embedding for the attribute is generated by calculating a weighted sum of the attributes in the attribute set.

9. A system comprising: one or more memory devices storing instructions; and One or more data processing devices configured to interact with the one or more memory devices and, when executing the instructions, perform operations including: obtaining an entity-attribute candidate pair defining an entity and an attribute, wherein the attribute is a candidate attribute of the entity; and Based on a sentence set including the entity and the attribute, determining whether the attribute is an actual attribute of the entity in the entity-attribute candidate pair, the determining comprising: generating embeddings for words in the set of sentences including the entity and the attribute, generating an attribute distribution embedding for the entity using known entity-attribute pairs, wherein the attribute distribution embedding for the entity specifies an embedding for the entity based on other attributes associated with the entity from the known entity-attribute pairs, generating an attribute distribution embedding for the attribute using the known entity-attribute pair, wherein the attribute distribution embedding for the attribute specifies an embedding for the attribute based on known attributes associated with a known entity for the attribute in the known entity-attribute pair, and Based on the embeddings for the words in the sentence set, the attribute distribution embeddings for the entities, and the attribute distribution embeddings for the attributes, it is determined whether the attributes in the entity-attribute candidate pairs are actual attributes of the entities in the entity-attribute candidate pairs.

10. The system according to claim 9, wherein: Generating the embedding for the word in the set of sentences including the entity and the attribute comprises: generating a first vector representation specifying a first embedding of a word between the entity and the attribute in the set of sentences; generating, based on the set of sentences, a second vector representation specifying a second embedding for the entity; and Based on the set of sentences, a third vector representation specifying a third embedding for the attribute is generated.

11. The system of claim 10, wherein: Generating the attribute distribution embedding for the entity includes: generating a fourth vector representation using the known entity-attribute pairs, the fourth vector representation specifying the attribute distribution embedding for the entity; and Generating the attribute distribution embedding for the attribute includes generating a fifth vector representation using the known entity-attribute pairs, the fifth vector representation specifying the attribute distribution embedding for the attribute.

12. The system according to claim 11, wherein: Determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair comprises: Based on the first vector representation, the second vector representation, the third vector representation, the fourth vector representation and the fifth vector representation, it is determined whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair.

13. The system according to claim 12, wherein: The operation of determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair is performed using a feed-forward network.

14. The system according to claim 13, wherein: Determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair comprises: combining the second vector representation, the third vector representation, the fourth vector representation, and the fifth vector representation into a single vector representation; inputting the single vector representation into the feed-forward network; and By means of the feed-forward network and using the single vector representation, it is determined whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair.

15. The system according to any one of claims 11 to 14, wherein: Generating the fourth vector representation comprises: identifying a set of attributes associated with the entity in the known entity-attribute pair, wherein the set of attributes does not include the attribute; and The attribute distribution embedding for the entity is generated by calculating the weighted sum of the attributes in the attribute set.

16. The system according to any one of claims 11 to 14, wherein: Generating the fifth vector representation comprises: Using the attributes, identifying a set of entities from the known entity-attribute pairs; For each entity in the set of entities, identifying a set of attributes associated with the entity, wherein the set of attributes does not include the attribute; and The attribute distribution embedding for the attribute is generated by calculating a weighted sum of the attributes in the attribute set.

17. A non-transitory computer readable medium storing instructions, which, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: obtaining an entity-attribute candidate pair defining an entity and an attribute, wherein the attribute is a candidate attribute of the entity; and Based on a sentence set including the entity and the attribute, determining whether the attribute is an actual attribute of the entity in the entity-attribute candidate pair, the determining comprising: generating embeddings for words in the set of sentences including the entity and the attribute, generating an attribute distribution embedding for the entity using known entity-attribute pairs, wherein the attribute distribution embedding for the entity specifies an embedding for the entity based on other attributes associated with the entity from the known entity-attribute pairs, generating an attribute distribution embedding for the attribute using the known entity-attribute pair, wherein the attribute distribution embedding for the attribute specifies an embedding for the attribute based on known attributes associated with a known entity for the attribute in the known entity-attribute pair, and Based on the embeddings for the words in the sentence set, the attribute distribution embeddings for the entities, and the attribute distribution embeddings for the attributes, it is determined whether the attributes in the entity-attribute candidate pairs are actual attributes of the entities in the entity-attribute candidate pairs.

18. The non-transitory computer-readable medium of claim 17, wherein generating the embedding for the word in the set of sentences including the entity and the attribute comprises: generating a first vector representation specifying a first embedding of a word between the entity and the attribute in the set of sentences; generating, based on the set of sentences, a second vector representation specifying a second embedding for the entity; as well as Based on the set of sentences, a third vector representation specifying a third embedding for the attribute is generated.

19. The non-transitory computer readable medium of claim 18, wherein: Generating the attribute distribution embedding for the entity includes: generating a fourth vector representation using the known entity-attribute pairs, the fourth vector representation specifying the attribute distribution embedding for the entity; and Generating the attribute distribution embedding for the attribute includes generating a fifth vector representation using the known entity-attribute pairs, the fifth vector representation specifying the attribute distribution embedding for the attribute.

20. The non-transitory computer readable medium of claim 19, wherein: Determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair comprises: Based on the first vector representation, the second vector representation, the third vector representation, the fourth vector representation and the fifth vector representation, it is determined whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair.

21. The non-transitory computer readable medium of claim 20, wherein: The operation of determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair is performed using a feed-forward network.

22. The non-transitory computer readable medium of claim 21, wherein: Determining whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair comprises: combining the first vector representation, the second vector representation, the third vector representation, the fourth vector representation, and the fifth vector representation into a single vector representation; inputting the single vector representation into the feed-forward network; and By means of the feed-forward network and using the single vector representation, it is determined whether the attribute in the entity-attribute candidate pair is an actual attribute of the entity in the entity-attribute candidate pair.

23. The non-transitory computer readable medium of any one of claims 19-22, wherein: Generating the fourth vector representation comprises: identifying a set of attributes associated with the entity in the known entity-attribute pair, wherein the set of attributes does not include the attribute; and The attribute distribution embedding for the entity is generated by calculating the weighted sum of the attributes in the attribute set.

24. The non-transitory computer readable medium of any one of claims 19-22, wherein: Generating the fifth vector representation comprises: Using the attributes, identifying a set of entities from the known entity-attribute pairs; For each entity in the set of entities, identifying a set of attributes associated with the entity, wherein the set of attributes does not include the attribute; and The attribute distribution embedding for the attribute is generated by calculating a weighted sum of the attributes in the attribute set.

Images