Specific ordering of text elements and their application

Abstract

Ranking a plurality of text elements according to specificity, each text element comprising at least one word. For each text element to be ranked, the method comprises: computing an embedding vector that positions the text element in an embedding space; and selecting a set of text snippets from a reference text. Each of the text snippets contains the text element to be ranked and a further text element. For each text snippet, the method computes a respective distance between the further text element in the embedding space. The method further comprises computing a specificity score for the text element to be ranked and storing the specificity score. After ranking the plurality of text elements, a text data structure can be processed to extract data having a desired specificity from the data structure using the specificity scores of the text elements.

Description

BACKGROUND

[0001] The present invention relates generally to the specific ordering of text elements. A computer-implemented method for ordering a plurality of text elements by specificity is provided, as well as applications of such a method. Systems and computer program products implementing these methods are also provided.

[0002] The specificity of a text element, such as a word or phrase, is a measure of the amount of information contained in those elements. When a text element contains a large amount of information in a given domain, the element is highly specific to that domain, and vice versa. The specificity of text has been estimated in the context of search systems to assess whether to return general or specific search results for a search query, or to suggest alternative search queries to a user. Most existing techniques for estimating specificity use statistics based on analyzing parts of speech, such as the frequency of noun modifications, or the frequency of occurrence of specific terms. One technique uses different measures derived from the vectors that locate those items in an embedding space generated via a word embedding scheme to assess the specificity of an item. This technique employs measures obtained by analyzing the distribution of embedding vectors of pre-trained embeddings. Once the embedding matrix is trained, the distribution of vectors in the embedding space is the only factor used to assess specificity. SUMMARY

[0003] One aspect of the invention provides a computer-implemented method for ordering a plurality of text elements by specificity, each text element comprising at least one word. For each text element to be ordered, the method comprises computing, by means of a word embedding scheme, an embedding vector that locates the text element in an embedding space, and selecting a set of text snippets from reference text. Each of these text snippets contains the text element to be ordered and a further text element. For each text snippet, the method computes a respective distance between the further text element and the text element to be ordered in the embedding space, each text element being located in said space by means of an embedding vector computed via the word embedding scheme. The method further comprises computing a specificity score for the text element to be ordered from the aforementioned distances, and storing the specificity score. The resulting specificity scores of the plurality of text elements define an ordering of the text elements by specificity.

[0004] A respective further embodiment of the invention provides a computing system adapted to implement the method of ordering text elements as described above, as well as a computer program product comprising a computer-readable storage medium embodying program instructions executable by a computing system to cause the computing system to implement such a method. BRIEF DESCRIPTION OF DRAWINGS

[0005] Embodiments of the invention will be described in more detail below by way of example and with reference to the accompanying drawings.

[0006] Figure 1 is a schematic diagram of a computing system for implementing the method of the present application;

[0007] Figure 2 show the component modules of a system embodying the present application for ranking text elements by specificity;

[0008] Figure 3 show the steps of the ranking method performed by the system shown in Figure 2

[0009] Figure 4 show the component modules of a text element ranking system in an embodiment of the present application;

[0010] Figure 5 show the steps of the word embedding process in the system shown in Figure 4

[0011] Figure 6 is a schematic diagram of the word embedding process;

[0012] Figure 7 show the operation of the context snippet selector in the system shown in Figure 4

[0013] Figure 8 show the steps of the specificity score computation process in the system shown in Figure 4

[0014] Figure 9A and 9B show the specificity ranking obtained in the implementation of the system shown in Figure 4

[0015] Figures 10 to 13 indicate the operational steps of an application using the text element ranking method embodying the present application. DETAILED DESCRIPTION

[0016] Some embodiments of the application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0017] ​​​​​A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or raised structures in

[0018] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0019] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0020] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0021] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0022] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0023] The computer program product of the second aspect of the application can comprise a computer readable medium to store the program code of the computer program of the first aspect of the application. The flow and block diagrams in the drawings represent architectures, functional and operational, of possible implementations of systems, methods and computer program products according to different embodiments of the application. In this regard, each block in the flow or block diagrams can represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by dedicated hardware-based systems that perform the specified functions or acts, or combinations of dedicated hardware and computer instructions.

[0024] The embodiments to be described can be implemented as a computer-implemented method for ranking text elements by specificity. Such a method can be implemented by a computing system comprising one or more general- or special-purpose computers, each of which can comprise one or more (real or virtual) machines providing functionality for implementing the operations described herein. The steps of the method embodying the application can be implemented by program instructions, e.g., program modules executed by processing devices of the system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. The computing system can be implemented in a distributed computing environment, such as a cloud computing environment, in which tasks are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0025] Figure 1 is a block diagram of an exemplary computing device for implementing the method embodying the application. The computing device is shown in the form of a general-purpose computer 1. The components of computer 1 can include a processing device such as one or more processors represented by processing unit 2, a system memory 3, and a bus 4 that couples various system components including system memory 3 to processing unit 2.

[0026] Bus 4 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EiSA) bus, Video Electronics Standards Association (vESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0027] The computer 1 typically includes a variety of computer readable media. Such media can be any available media that is located either internally or externally to computer 1, such as volatile and non-volatile media, and removable and non-removable media. By way of example, system memory 3 can include computer readable media in the form of volatile memory, such as random access memory (RAM) 5 and / or cache memory 6. Computer 1 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 7 can be provided for reading from and writing to non-removable, non-volatile magnetic media (typically called a "hard drive"). Although not explicitly shown, other hardware and / or software components could also be used, such as a mass disk controller, a disk controller, a tape controller, and / or a network controller, to name a few. As mentioned, storage system 7 can include a hard disk drive (HDD) 9. As used herein, the term "hard disk drive" refers to any data storage device that uses one or more non-removable, non-volatile magnetic elements to store data. The HDD 9 can be connected to the system bus 4 by an HDD interface 8. As mentioned, the storage system 7 can also include a magnetic floppy disk drive (FDD) 10 to change the HDD 9. As used herein, the term "floppy disk drive" refers to any device capable of accepting a removable, non-removable, and / or non-volatile magnetic disk, such as a "floppy disk" or "floppy diskette", and which is capable of reading data from and / or writing data to the magnetic disk. In this context, a magnetic disk drive can be considered a computer system storage medium. A magnetic disk interface can be used to provide a data media with a computer system. The disk drive FDD 10 can be connected to the system bus 4 by a FDD interface 11. Although the exemplary embodiment describes a magnetic disk drive, those skilled in the art will appreciate that other types of computer system storage media which are readable by a computer system can also be used. In addition, those skilled in the art will appreciate that a computer system can have one or more computer system storage media connected thereto. The storage system 7 stores software elements, shown as being stored in the storage system 7, includes an operating system 8, one or more application programs 9, other program modules 10, and program data 11. This basic configuration is for purposes of illustration and not limitation. Those skilled in the art can employ other computer systems, effectively adapted to execute embodiments of the present application, without departing from the scope thereof.

[0028] The memory 3 can include at least one program product having one or more program modules that are configured to carry out the functions of embodiments of the application. By way of example, the program / utility 8, having a set (at least one) of program modules 9, can be stored in memory 3. The program / utility 8, operating system, one or more application programs, other program modules, and program data, can include an implementation of the network environment. The program modules 9 typically execute the functions and / or methodologies of embodiments of the application as described herein.

[0029] The computer 1 can also communicate with one or more external devices 10 such as a keyboard, a pointing device, a display 11, etc.; one or more devices that enable a user to interact with computer 1; and / or any devices (e.g., network card, modem, etc.) that enable computer 1 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 12. Still yet, computer 1 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 13. As depicted, network adapter 13 communicates with the other components of computer 1 via bus 4. Computer 1 can also communicate with additional processing devices 14 such as GPUs (graphics processing units) or FPGAs to implement embodiments of the present application. It will be appreciated that, although not shown, other hardware and / or software components that can be used in conjunction with computer 1. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0030] Figure 2 The diagram illustrates the component modules of an exemplary computing system embodying the present application. The system 20 includes a memory 21 and control logic, generally indicated at 22, including functionality for ranking text elements by specificity. The control logic 22 includes a word embedding module 23, a context selector module 24, and a specificity calculator module 25. Each of these modules includes functionality for implementing particular steps of the ranking process detailed below. These modules interface with the memory 21, which stores different data structures used in the operation of the system 20. These data structures include a set of N text elements 27 (here represented by {t i}, i = 1 to N) to be ranked by specificity, a set of embedding vectors 28 generated by the word embedding module 23, and a set of text snippets (“context snippets”) 29 selected by the context selector 24 from a reference text represented in the diagram by a text corpus 30. A set of specificity scores 31 (here represented by {S i}, i = 1 to N) generated by the specificity calculator 25 for the text elements {t i}, i = 1 to N) are also stored in the system memory 21.

[0031] Generally, the functionality of the logic modules 23-25 can be implemented by software (such as program modules) or hardware or a combination thereof. In other embodiments, the described functionality can be allocated differently among the system modules, and the functionality of one or more modules can be combined. The component modules of the system 20 can be provided in one or more computers of a computing system. For example, all of the modules can be provided in a computer 1, or the modules can be provided in one or more computers / servers to which user computers can connect via a network (which can include one or more component networks and / or the Internet, including the Internet) for inputting text items to be ranked. The system memory 21 can be implemented by one or more memory / storage components associated with one or more computers of the system 20.

[0032] The set of text elements {t i} can contain individual words and / or multiword expressions (MWEs), and can be compiled for a particular application / domain or span multiple domains for different applications. Some embodiments of the present application incorporate MWEs in {t i} to exploit the inherent specificity of these elements. A list of MWEs {t i} can be precompiled manually or automatically for storage in the system memory 21 as described below.

[0033] The reference text corpus 30 can be local or remote to the system 20, and can include text from across the elements {t itext of one or more information sources of the domain of the reference text corpus. While represented in Figure 2

[0034] Figure 3 The flowchart of Figure 2 indicates the steps of the ranking process performed by the system 20 (designated as step 34 of ranking the text elements). Step 35 represents storing a set of text elements {t i} to be ranked in the system memory 21. In step 36, the word embedding module 23 computes an embedding vector for each text element t i via a word embedding scheme. Word embedding schemes are well known and in essence generate a mapping between text elements and vectors of real numbers that define the position of the respective text element in a multi-dimensional embedding space. The relative position of text elements in this space indicates the degree of relationship between the text elements, with elements that are "closer" in the embedding space being more closely related than elements that are further apart. In particular, the idea of word embeddings is to map elements that occur in similar textual contexts to be close to each other in the embedding space. Any desired word embedding scheme can be employed here, including context- independent schemes using, for example, Word2Vec, GloVe (Global Vectors) or FastText models, or context-dependent schemes including models based on transformer architectures, such as the BERT (Bidirectional Encoder Representations from Transformers) model. The word embedding scheme can generate a "cloud" of text elements that occur in similar contexts and thus represent semantically similar concepts. Each of the embedding vectors generated by the word embedding module 23 thus locates the corresponding text element t i in the embedding space, here denoted by χ. The resulting vectors are stored in the system memory 21 at 28.

[0035] In step 37, the context selector module 24 selects a set of text snippets from the reference text 30 for each element t i to be ranked. Each of these text snippets contains the element t i and further text elements (where these further text elements can or can not include one or more of the other elements t i to be ranked). For example, the context selector 24 can select a text snippet containing the element t i or a given element t i ​The surrounding text window contains lines of text, sentences, or paragraphs. Typically, a collection of one or more text segments containing a given text element can be selected here; some embodiments select multiple segments for each element. The selected text segments (possibly after further processing described below) are stored as context segments 29 in system memory 21.

[0036] Steps 38 to 40 instruct the operation of the specificity calculator 25. For each text element t to be sorted... i These steps are performed. In step 38, the specificity calculator retrieves from the context fragment set 29 the content containing the given element t. i The context fragment. For each fragment, the specificity calculator 25 calculates the element t in the embedding space χ. i The corresponding distances between the text element and other text elements in the segment. To calculate these distances, each further text element must first have its corresponding embedding vector, calculated via a word embedding scheme, located in the embedding space χ. The embedding vectors for the further text elements can be pre-calculated in step 36 above, for example, via a context-independent embedding scheme detailed below, or can be dynamically calculated via a context-dependent embedding scheme. It is convenient to use t i The χ² distance between elements and other text elements is calculated as the cosine similarity between two vectors representing those elements. However, in other embodiments, any convenient distance metric, such as Euclidean distance, can be used. In step 39, the specificity calculator calculates the element t. i Specificity score. Element t i Specificity score S i This depends on the distance calculated in step 38 from the context fragment containing the element. The specificity score can be calculated from these distances in various ways as explained below. In step 40, the specificity score S is... i Stored in group 31 of system memory 21. After processing all text elements {t} i Following the context fragment of}, a set of specificity scores {S} is generated. i Then, the order of these text elements is defined by specificity.

[0037] The above approach adapts the context of a text element when computing specificity by using distances in the embedding space χ between text elements in a context snippet. By injecting context into the information extracted from word embeddings, the resulting specificity scores provide a measure of homogeneity of the context in which a text element appears. This provides a true measure of specificity on the basis that highly specific terms tend to appear in more homogeneous backgrounds than more general terms. As an illustrative example, the term "information hiding" is a highly technical expression used in software engineering when a data structure is designed not to expose internal state to the outside. In contrast, "hiding information" is a term that can be used in many different contexts, and therefore will appear. The above techniques therefore provide an improved estimate of specificity, with consequent advantages for performance and resource efficiency of information extraction applications. The techniques are also fully unsupervised, allowing specificity scores to be computed for any set of text elements without requiring annotated training data.

[0038] Figure 4 A more detailed system implementation in some embodiments of the invention is shown. The system 45 of this embodiment is adapted to compile and rank a large set of MWEs represented by {m i}. The control logic 46 of this system includes a word embedding module 47, a context selector 48, and a specificity calculator 49 as previously described. The control logic also includes an MWE extractor module 50 and a text encoder module 51. Data structures stored in system memory 53 include a set of MWEs {m i} 54 and a tokenized text dataset 56, where the set of MWEs {m i} 54 is compiled automatically by the MWE extractor 50 from a knowledge base indicated schematically at 55, and the tokenized text dataset 56 is generated by the text encoder 51 from a text corpus indicated as WE (word embedding) corpus 57. In practice, the knowledge base 55 and the WE corpus 57 can represent content collated from or distributed across multiple information sources. The memory 53 also stores an embedding matrix 58 generated by the word embedding module 47, and a set of inverse frequencies 59 described further below. In addition, the memory 53 stores a set of context snippets 60 generated by the context selector 48, and a set of example scores 61 described further below, and a final set of specificity scores 62 computed for the MWEs {m i}.

[0039] The operation of the system 45 will be described with reference to Figures 5 to 8 . Figure 5Flowchart 500 illustrates the operational steps leading to the generation of embedding matrix 58. In step 65, MWE extractor 50 accesses knowledge base 55 to extract MWEs associated with hyperlinks in the knowledge base. Knowledge bases (such as Wikipedia, DBPedia, Yago, etc.) are essentially graphs of concepts, where concepts are linked to each other. MWE extractor 50 can extract MWEs from the knowledge base by searching for hyperlinks. For example, in the sentence (where hyperlinks are indicated by underscores): “In a thermal power plant, mechanical power is generated by a thermal engine that converts the heat energy from the combustion of fuel into rotational energy,” the MWE extractor can select “thermal engine” and “heat energy.” Hyperlinks in such knowledge bases are manually annotated and therefore of high quality. By simply scanning the knowledge base text, MWE extractor 50 can extract a large number of well-defined MWEs. In this example, the MWE extractor searches knowledge base 55 to compile a large dictionary of MWEs covering a wide range of topics. In step 66, the resulting set of MWEs 54 {m i It is stored in memory 53.

[0040] In steps 67 and 68, the text encoder 51 generates tokenized text 56 by preprocessing and tokenizing the WE corpus 57, so that each MWE m i Each word in the corpus is encoded as a single token, and other words in the corpus are encoded as corresponding tokens in the tokenized text. Specifically, in step 67, the text encoder preprocesses the WE corpus 57, as shown in... Figure 6 The diagram 600 illustrates the data stream. Instances of MWE were identified in the original corpus, and each of these was concatenated and treated as a separate word. For example, the MWE “machine learning” was concatenated as “machine_learning”. During preprocessing, all units and stop words (such as “a”, “and”, “was”, etc.) were removed, and all uppercase letters were converted to lowercase. The resulting text was then segmented into sentences for training word embeddings. Figure 5 In step 68, the preprocessed text is tokenized by encoding all remaining words and MWE into corresponding single tokens. While other encoding schemes are certainly conceivable, hot encoding is conveniently employed here. Each token thus represents a specific word / MWE, and that word / MWE is replaced by the corresponding token wherever it appears in the preprocessed text. The resulting tokenized text 56 is stored in system memory 53.

[0041] In step 69, the word embedding module 47 processes the tokenised text 56 to produce an embedding matrix 58. In this embodiment, the tokenised sentences in the text 56 are used to train a Word2vec embedding model using known CBOW (Common Bag Of Words) and negative sampling techniques (see, for example, “Distributed representation of words and stems and their compositionality,” Mikolov et al., Advances in Neural Information Processing Systems 26, 2013, pp. 3111-3119). This results in a set of embedding vectors, one for each of the tokens corresponding to the respective MWEs / words in the pre-processed text, as shown schematically in Figure 6 In step 70, this set of vectors constitutes the embedding matrix 58 stored in the system memory 53. The resulting embedding matrix therefore includes embedding vectors corresponding to the text elements (here, MWEs) to be ordered, as well as further text elements to be included in the text snippets selected by the context selector 48 from the reference text corpus 30. (In this regard, while Figure 4 different reference text corpus 30 is shown in

[0042] When processing the WE corpus 57, the embedding module 47 counts the number of instances of each text element (MWE or word, here generally denoted w) in the pre-processed corpus. For each element w, the embedding module calculates the inverse frequency of occurrence of that element, f(w). The inverse frequency of occurrence of an element w that occurs n times in a corpus of m wordpieces is defined as f(w) = m / n. At 59, the set of inverse frequencies f(w) for the elements w is stored in the system memory 53.

[0043] The operation of the context selector 48 is indicated here in the dataflow of Figure 7 In this example, the context selector 48 uses a different reference text corpus 30 to the WE corpus 57. In Figure 7 In step (a) of i all instances of the MWE m iEach processed sentence of an instance of the corpus is selected as a context fragment. The context selector then stores each context fragment in a set 60, here as a "Bag of Words" (BOW), as indicated by step (d).

[0044] Figure 8 The flowchart 800 illustrates the operation of the specificity calculator 49 in this embodiment. In step 75, the specificity calculator selects a context fragment from the set 60 of fragments. It then computes, for the selected fragment, a distance d(m i between each MWE m i and each further text element (MWE / word) w in the BOW for which the embedding vector is contained in the embedding matrix 58. The distance, denoted here by d(m i ,w), is computed as the cosine similarity between the two vectors representing m i and w, respectively. This yields a number in the range (-1, +1), where a higher number indicates closer elements m i and w in the embedding space χ.

[0045] The specificity calculator then computes, from the distances computed in step 76 for the current fragment, an instance score 61 for the MWE m i of the instance. In this embodiment, each distance d(m i ,w) is first weighted based on the inverse occurrence frequency f(w) stored in the set 59 for the element w, and the instance score for the MWE m i is computed from the weighted distances of the fragment. Specifically, in step 78, the specificity calculator obtains the instance score by aggregating the weighted distances of the fragment. In this instance, given a BOW comprising the MWE m i and further elements w1,..., w k , the instance score T i is computed as:

[0046]

[0047] If there are more context fragments to be processed in the determination step 79, the operation returns to step 75, in which the next fragment is selected from the set 60 and processed as described above. When all context fragments have been processed in step 79, the operation proceeds to step 80. Here, for each MWE m i , the specificity calculator 49 computes a specificity score S i as the instance score T i for m iThe function. In this embodiment, the aggregated instance score T is used. i To calculate the specificity score S i Here, S is used as a simple average. i =∑T i Therefore, the final specificity score S of MWE i It is the instance score T of MWE aggregated over the entire reference corpus. i Corpus-level scores. Specificity scores {S} obtained. i Group 62 analyzed all MWE m in the WE corpus 57 using specificity. i Sort them.

[0048] The inverse frequency-based distance weighting penalizes the contribution of common (and possibly more general) elements w, thus biasing the average towards less common (and possibly more specific) elements. By generating the embedding matrix 58 from a large and diverse WE corpus 57 with a large MWE dictionary, the system described above can automatically generate specificity scores for use in a wide range of applications. However, typically, specificity scores {S} can be computed for any subset of tokens in the embedding space χ, for the MWE, and / or for individual words. i Furthermore, this subset can be specific to a given domain or application. In other embodiments, an embedding matrix 58 can also be generated for MWE / words related to a specific technology / application.

[0049] Figures 9a (900a) and 9b (900b) illustrate the process from... Figure 4 Extraction of specific sorting generated by the system implementation method. Figure 9A The results were obtained using a corpus of reference texts that includes 1.5 million patent abstracts. Figure 9B The results were obtained using a reference text corpus that includes 12 million abstracts from arXiv papers. Both sets of results used embedding matrices built from the WE corpus, which contains more than 100 million news articles. Figure 9A The specificity scores of the 10 highest and lowest scores for the MWE containing the word "knowledge" are shown. Figure 9b shows the 10 highest and lowest scores for the MWE containing the word "language," which can be seen as scores that correlate well with the specificity of the listed MWEs. These examples demonstrate that the specificity scores calculated using the techniques described above, or even as a simple average of instance scores calculated on a large reference corpus, can reliably distinguish highly technical MWEs from more common expressions.

[0050] Particularity ranking techniques can be used to enhance the operation of many data processing applications, where, after ranking text elements by particularity, the particularity scores are used in processing the text data structure to extract data having a desired particularity. Using the particularity scores can reduce the processing resources required to extract relevant data from different data structures for different purposes, and can improve the quality of the extracted data, thus enhancing the performance of the applications using these data structures. See below Figures 10 to 13 Some illustrative applications are described.

[0051] Figure 10 Flowchart 1000 of FIG. 1 illustrates the operation of a knowledge induction system for extracting knowledge from a large corpus of text. Such an application typically processes a large amount of text mined from databases / websites in the cloud. Step 85 represents the storage of the cloud data to be analyzed. In step 86, the ranking method described above is employed to rank the text elements in the data by particularity. In step 87, the cloud data is filtered based on the particularity scores to identify a set of the most particular text elements in the corpus, e.g., the text elements having a particularity score greater than a defined threshold. In step 88, a knowledge graph (KG) is then constructed from the filtered data. Knowledge graphs are well-known data structures that are often used to extract meaningful knowledge from large amounts of data for industrial, commercial or scientific applications. A knowledge graph primarily comprises nodes representing entities interconnected by edges representing relationships between the connected entities. The knowledge graph constructed in step 88 thus comprises nodes corresponding to the elements in the set of identified most particular text elements, interconnected by edges representing relationships between those nodes. (As will be apparent to those skilled in the art, such relationships can be defined in various ways for a particular application.) The resulting knowledge graph provides a data structure that can be searched to extract the information represented in the graph. In response to an input search query in step 89, the system then searches the graph in step 90 to extract the requested data from the data structure. The filtering of the data used to construct the knowledge graph in this application can significantly reduce the size of the data structure, and thus the memory required to store the graph, while ensuring that the most particular data containing the most information is retained. The computational intensity of the search operation is likewise reduced, and the search results focus on more specific, often more useful information.

[0052] Another application of the particularity scores involves the expansion of a set of keywords for a search process. Figure 11 Flowchart 1100 of FIG. 1 illustrates the operation of such a system. Step 95 represents the storage in the system of a matrix of word embeddings comprising vectors positioning corresponding text elements in a latent embedding space. Such a matrix can be constructed in the manner described above with respect to Figure 4In a similar manner to the embedding matrix 58, the embedding matrix 60 is generated and can encode a wide range of words / MWEs in one or more technical domains. In step 96, the text elements in the embedding matrix are ordered by specificity as described above. Step 97 represents the user inputting a keyword related to the field to be searched, which is represented by a vector in the embedding matrix. In step 98, the system then searches the embedding space around this keyword to identify neighboring text elements in the embedding space. Different clustering / nearest neighbor search processes can be employed here, where the search process is adapted to locate a set of most specific text elements (e.g., elements with specificity scores above a desired threshold) that are adjacent to the input keyword. In step 99, the text elements so identified are stored with the user input keyword as an expanded keyword set. This expanded keyword set can then be used to search a corpus of text, e.g., by string matching the keywords in the set against documents in the corpus, to identify relevant documents in the desired field. The use of specificity scores in this application allows for the automatic expansion of small, user input keyword sets using highly specific, relevant keywords, facilitating the location of relevant documents in a given domain. A particular example of this application is for the curation of training documents for training a text classifier model.

[0053] Flowchart 1200 Figure 12 The use of specificity scores in an automated phrase extraction system is illustrated. Phrase extraction systems are well known and can be used to extract topic phrases or key phrases from documents for abstraction / generalization purposes (see, e.g., “Key2Vec: Automatic ranked keyphrase extraction from scientific articles using phrase embeddings,” Mahata et al., Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2, June 2018, pp. 634-639). These systems often use a graph-based representation for candidate key phrases in a document. Nodes representing candidate phrases are interconnected by edges (representing relationships between the nodes) with associated weights (depending on semantic similarity, frequency of occurrence, etc.), which are then used to select the best candidates. Figure 12Step 100 of Figure 1 represents the usual text processing operations for generating graphs of candidate phrases. In step 101, the text elements in the graph are ordered by specificity using the method described above. In step 102, the graph is pruned to obtain a subgraph representing the most specific subset of the phrases in terms of the specificity scores of the text elements in the candidate phrases. This most specific subset can include phrases containing text elements with specificity scores above a desired threshold. In step 103, the resulting subgraph is then processed in the usual way to extract the best candidate phrases from the subgraph. Such processing can involve scoring the nodes based on different graph features to extract the best phrases for a desired purpose.

[0054] Figure 13 Figure 1300 of the flowchart 1300 illustrates the use of specificity scores in a search system. In step 105, the text elements in a search database are ordered by specificity as described above. In response to the input of a search query in step 106, the system identifies any ordered text elements in the query text. In step 108, the system generates a response to the search query by retrieving data from the search database according to the specificity scores of any ordered text elements so identified. The response here can be to suggest alternative search queries to the user, or to retrieve the requested data from the search database. The specificity scores can be used here to identify the most relevant alternative queries or response data based on the elements with the highest specificity scores in the input query. The specificity scores can also be used to assess the knowledge level of the user and return results accordingly. For example, an input query containing highly specific text elements suggests a knowledgeable user wants more detailed results, while a low-specificity query suggests the user needs more general high-level results.

[0055] As can be seen, the specificity ordering techniques provide more efficient processing and improved results in a variety of processing applications, and can reduce the memory and processing resources required for knowledge extraction operations.

[0056] The methods embodying the invention are based on the recognition that highly specific text elements, such as those representing technical concepts, tend to occur in text contexts that are homogenous in nature. These methods use a snippet of reference text to provide context for a text element to be ordered. The specificity score for a given text element is then based on the distance in embedding space between that text element and other text elements in the selected text snippet containing that element. The methods embodying the invention build on the foregoing implementation to incorporate the context of the text element in the information extracted from the word embeddings, so that the resulting specificity scores provide a measure of the homogeneity of the context in which the text element occurs. This provides an elegant and simple technique for capturing the specificity of a text element, providing an improved estimate of specificity, and enhanced performance of processing systems using such estimates.

[0057] After sorting the plurality of text elements, the method embodying the present invention can process the text data structure using the specificity scores for the text elements to extract data having a desired specificity from the data structure. Using the specificity scores can reduce the processing resources required to extract relevant data from the data structure in different applications, and can improve the quality of the extracted data, thus enhancing performance. For example, the specificity scores can be used as a filtering mechanism to reduce the memory required to store a search structure, such as a knowledge graph, for example, by pruning the graph to remove unnecessary elements, thus reducing the computational intensity of search operations performed on such graphs. Examples of other text data structures and processing applications employing these structures will be described in more detail below.

[0058] Generally, the text elements to be sorted can include word text elements (i.e., individual words) and / or multi-word expressions (i.e., text elements including at least two words). Multi-word expressions include combinations of words, such as open compound words or phrases, that collectively convey a particular meaning or act as a semantic unit at some level of linguistic analysis. In some embodiments, the plurality of text elements to be sorted includes multi-word expressions, taking advantage of the fact that they are generally inherently more specific than individual words. A single embedding vector is then computed for each multi-word expression, i.e., the multi-word expression is treated as if it were a single word for the embedding process. If desired, the text elements to be sorted can of course include individual words as well as multi-word expressions.

[0059] Some embodiments of the present invention select, from a reference text, a plurality of text segments containing each text element to be sorted. For each text segment (e.g., sentence) containing an element to be sorted, these embodiments compute an instance score that depends on the distance between the text element and other text elements in the segment. The specificity score for the text element is then computed from the instance scores of the plurality of text segments containing the element. The accuracy of the specificity score generally improves with an increase in the number of text segments selected as context for the text element. In some embodiments of the present invention, the reference text comprises a text corpus, and for each text element to be sorted, segments of the text corpus are selected for each instance of the text element in the corpus.

[0060] In computing the instance score from the text segment, some embodiments weight the distance between the text element to be sorted and each additional text element by the inverse frequency (explained below) of occurrence of the additional text element in the text corpus (e.g., the corpus used to compute the embedding vectors). The instance score is computed as a function of these weighted distances for the segment. The weighting serves to penalize the contribution of more common words, giving more weight to infrequent words, thus enhancing the accuracy of the specificity score.

[0061] The embedding vectors can be computed by any convenient word embedding scheme, which can include a context-independent or context-dependent embedding scheme. A context-independent word embedding scheme processes a corpus of text to produce an embedding matrix containing embedding vectors for selected text elements (here, words and / or multiword expressions) in that text. A context-dependent scheme employs an embedding model that can take any input text and output an embedding vector for that text. Embodiments that use context-independent embeddings have been found to provide enhanced precision in specificity computation, especially for more technical terminology. Accordingly, certain methods process a corpus of text to generate an embedding matrix. In particular, some embodiments symbolize a corpus of text so that, in the tokenized text, each of the text elements to be ordered is encoded as a single symbol, and other words in the corpus are encoded as respective symbols. The symbolized text is then processed by a word embedding scheme to generate an embedding matrix that includes embedding vectors corresponding to the text elements to be ordered and further text elements to be extracted from text segments selected for context purposes. A set of multiword expressions to be encoded as a single token can be stored prior to tokenization of the corpus. Some embodiments of the invention can compile a set of multiword expressions automatically by processing a set of text data (e.g., by automatically extracting expressions from a large set of documents, or by identifying hyperlinks containing multiword expressions in text from an online knowledge base). In this way, a large dictionary of multiword expressions can be compiled for use in the embedding process. All or a subset of these can then be ordered by specificity as desired.

[0062] Of course, it will be appreciated that many changes and modifications can be made to the described example embodiments. For example, in other embodiments, a lexicon of MWEs to be ordered can be extracted by an automatic phrase extraction system. Instance scores can be computed differently by averaging, summing, or otherwise aggregating distances or weighted distances, and specificity scores can be computed as other functions of instance scores or underlying distances. As an example, a specificity score can be based on a statistical treatment of the distribution of instance scores for the elements, such as a statistical mean after removing the highest and lowest instance scores from the distribution.

[0063] The steps of the flowcharts can be implemented in a different order than shown, and some steps can be performed in parallel where appropriate. In general, where features are described herein with reference to a method embodying the invention, corresponding features can be provided in a computing system / computer program product embodying the invention, and vice versa.

[0064] The description of various embodiments of the application has been presented for purposes of illustration but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technology found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for ranking a plurality of text elements, comprising: computing, by a word embedding scheme, embedding vectors that position, in an embedding space, a first text element of the plurality of text elements to be ranked; selecting, from a reference text, a set of text snippets, wherein each text snippet contains the first text element to be ranked and at least one other text element; for each text snippet, computing respective distances, in the embedding space, between the at least one other text element to be ranked and the first text element to be ranked, each other text element being positioned in the space by an embedding vector computed via the word embedding scheme; computing, from the respective distances in the embedding space, a specificity score for the text elements to be ranked; and storing the specificity score, wherein the specificity score of the plurality of text elements defines a ranking of the text elements by specificity. the plurality of text elements to be ranked comprises multi-word expressions.

2. The method of claim 1, wherein, the plurality of text elements to be ranked comprises single-word text elements.

3. The method of claim 1, wherein, 4. The method of claim 1, further comprising processing a text corpus by: tokenizing the text corpus such that each of the text elements to be ranked is encoded as a single token and other words in the text corpus are encoded as respective tokens; and processing the tokenized text by the word embedding scheme to generate an embedding matrix comprising the embedding vectors corresponding to the text elements to be ranked and the at least one other text element to be ranked.

5. The method of claim 4, further comprising: storing a set of multi-word expressions prior to tokenizing the corpus; and encoding each multi-word expression in the set of multi-word expressions as a single token at the same time as tokenizing the corpus.

6. The method of claim 5, further comprising compiling the set of multi-word expressions by processing a text dataset.

7. The method of claim 1, further comprising: for each text element to be ranked: selecting, from the reference text, a plurality of text snippets containing the first text element; for each text snippet, computing an instance score from a distance between the first text element to be ranked and the at least one other text element in the snippet; and computing the specificity score from the instance scores of the plurality of text snippets.

8. The method of claim 7, further comprising: for each text snippet: weighting the distance between the first text element to be ranked and each of the at least one other text element by an inverse frequency of occurrence of the other text element in a text corpus; and computing the instance score from the weighted distances of the text snippet.

9. The method of claim 8, further comprising: for each text element to be ranked: computing the instance score for the snippet by aggregating the weighted distances of each of the plurality of text snippets; and computing the specificity score by aggregating the instance scores of the plurality of text snippets. ​ 10. The method of claim 7, wherein, The reference text comprises a corpus of text; and For each text element to be ordered, a segment of the corpus of text is selected for each instance of the first text element in the corpus of text.

11. The method of claim 1, wherein, Each text segment comprises a sentence.

12. The method of claim 1, comprising: After ordering the plurality of text elements, the text data structure is processed using the specificity scores of the text elements to extract data of a desired specificity from the data structure.

13. The method of claim 12, wherein, The text data structure comprises a corpus of text, and wherein the method further comprises: using the specificity scores of the text elements in the corpus to construct a knowledge graph comprising a set of most specific text elements in the corpus; and in response to input of a search query, searching the knowledge graph to extract data relevant to the search query.

14. The method of claim 12, wherein, The text data structure comprises a word embedding matrix comprising vectors that position respective text elements in a latent space, and the method further comprises: in response to input of a text element corresponding to a vector in the latent space, identifying a set of most specific text elements adjacent to the input text element in the latent space based on the specificity scores.

15. The method of claim 12, wherein, The text data structure comprises a graph having nodes representing text phrases, the graph being interconnected by edges representing relationships between the nodes, and the method further comprises: pruning the graph to obtain a subgraph representing a most specific subset of the phrases according to specificity scores of the text elements in the text phrases; and processing the subgraph to extract the desired phrases.

16. The method of claim 12, wherein, The text data structure comprises a search database, and the method comprises: in response to input of a search query: identifying any ordered text elements in the search query; and generating a response to the search query by extracting data from the search database according to the specificity scores of any ordered text elements thus identified.

17. A computer program product for ordering a plurality of text elements, comprising: a machine readable storage device; and computer code stored on the machine readable storage device, wherein the computer code comprises instructions and data for causing a processor to perform any of the steps of any of the methods of claims 1-16.

18. A computing system for ordering a plurality of text elements, comprising: a processor; a machine readable storage device; and computer code stored on the machine readable storage device, the computer code comprising instructions and data for causing the processor to perform any of the steps of any of the methods of claims 1-16.

Images