An efficient short text similarity determination method and apparatus

Abstract

One aspect of the present disclosure relates to an efficient short text similarity determination method, including segmenting short texts in a corpus to obtain corresponding word sequences; determining a penalty based on a total number of short texts in the corpus, the penalty decreasing as the total number of short texts in the corpus increases; determining a term frequency of each word in the word sequences and an adjusted inverse document frequency, wherein the adjusted inverse document frequency is calculated based on the penalty; weighting the term frequency of each word with the adjusted inverse document frequency; combining the weighted term frequencies of each word in the word sequences to determine a term frequency vector of the short text; and determining a similarity of the short text with other short texts based on the term frequency vector. The present disclosure also relates to other related aspects.

Description

TECHNICAL FIELD

[0001] The present application generally relates to natural language processing (NLP), and in particular to efficient short text similarity determination. BACKGROUND

[0002] Text similarity measurement is a common problem in the field of NLP. For long texts and short texts, different measurement methods have been studied by the academic and industrial communities.

[0003] For long text similarity measurement methods, there are usually two paradigms: 1) vectorize words or phrases, aggregate the vector representation of long texts to calculate similarity, common ones are: Word2vec, Bow model, etc.; 2) introduce deep learning network structure, learn the vector of sentence or text according to the context semantics, common ones are Elmo, Bert, etc., calculate the similarity by constructing the sentence vector.

[0004] For short text similarity measurement methods, there are usually two classical paradigms: 1) do not vectorize the sentence, directly calculate the character-level similarity coefficient, common ones are jaccard similarity, Sorensen similarity coefficient, Levenshtein distance, and Hamming distance, etc.; 2) vectorize the sentence after segmentation, such as the classic one-hot, TF-idf algorithm, etc.

[0005] However, there are several problems in the similarity measurement of short texts, including: 1) character-level similarity measurement ignores the word order factor; 2) for large-scale corpus, the vector dimension of one-hot increases linearly, and the data sparsity leads to indistinct similarity, while for small-scale corpus, the calculation formula of TF-idf will cause the idf value of low-frequency words to be too large, and the text vector has convex points, and the similarity fluctuates greatly. Therefore, there is a need in the art for improved more efficient and accurate short text similarity determination techniques. SUMMARY

[0006] One aspect of the present disclosure relates to a short text similarity determination method, comprising segmenting short texts in a corpus to obtain corresponding word sequences; determining a penalty based on the total number of short texts in the corpus, the penalty decreasing as the total number of short texts in the corpus increases; determining the word frequency of each word in the word sequence and an adjusted inverse document frequency, wherein the adjusted inverse document frequency is calculated based on the penalty; weighting the word frequency of each word with the adjusted inverse document frequency; combining the weighted word frequency of each word in the word sequence to determine the word frequency vector of the short text; and determining the similarity of the short text with other short texts based on the word frequency vector.

[0007] According to some example embodiments, determining the adjusted inverse document frequency for each word includes determining a total number of texts in the corpus; determining a number of texts in the corpus that contain the word; adjusting the number of texts in the corpus that contain the word based on the penalty such that the number of texts in the corpus that contain the word is exponentially increased when a total number of short texts in the corpus is less than a first threshold; and determining the adjusted inverse document frequency based on the total number of texts and the adjusted number of texts in the corpus that contain the word.

[0008] According to some example embodiments, adjusting the number of texts in the corpus that contain the word based on the penalty further causes the number of texts in the corpus that contain the word to not be increased when a total number of short texts in the corpus is greater than a second threshold.

[0009] According to some example embodiments, the penalty includes an exponential smoothing factor, and the adjusting includes adding the number of texts in the corpus that contain the word to the exponential smoothing factor.

[0010] According to some example embodiments, weighting the term frequency for each word with the adjusted inverse document frequency includes multiplying the term frequency by the adjusted inverse document frequency calculated based on the penalty.

[0011] According to some example embodiments, the method further includes padding or cutting the sequence of words of the short text or the sequence of words of the other short text to make them have the same length if the sequence of words obtained by tokenizing the short text has a different length than the sequence of words of the other short text.

[0012] According to some example embodiments, determining the similarity of the short text to other short texts based on the term frequency vectors includes calculating a cosine distance between the term frequency vector of the short text and the term frequency vector of the other short text.

[0013] Other aspects of the disclosure also include apparatuses, devices, computer readable storage media, and the like, that implement the functionality of the respective methods. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A schematic diagram of a short text similarity determination system according to an aspect of the disclosure is shown.

[0015] Figure 2 A schematic diagram of a term frequency determination apparatus according to an aspect of the disclosure is shown.

[0016] Figure 3 A flowchart of a short text similarity determination method according to an aspect of the disclosure is shown.

[0017] Figure 4A block diagram of a short text similarity determination apparatus is shown in accordance with an aspect of the present disclosure. DETAILED DESCRIPTION

[0018] Term Frequency-Inverse Document Frequency (TF-IDF) technique is a commonly used weighting technique for data retrieval and text mining, which can be used to evaluate the importance of a single word to a certain text in a text library or corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, i.e., term frequency (TF), but decreases in inverse proportion to the frequency of its appearance in the corpus, i.e., inverse document frequency (IDF). If a word is relatively rare but appears multiple times in an article, it is likely to reflect the characteristics of the article and is the desired keyword.

[0019] To count the keywords of a text, the text can be segmented first, and then the term frequency of each word can be counted. The term frequency refers to the number of times a given word appears in the text. The keywords of a text have a high term frequency in the text. However, high-frequency words without significant meaning, such as “of”, “get”, “place”, “is”, “also”, etc., are easily mistaken for keywords. Therefore, each word needs to be weighted to reduce the weight of such high-frequency words without significant meaning, and to increase the weight of words that are less common in the average sense of the corpus but have significant meaning in the text. This weight can be referred to as inverse document frequency (IDF).

[0020] Inverse document frequency is a measure of the importance of a word and is inversely proportional to the degree of commonality of the word in general. For example, the inverse document frequency can be calculated by dividing the total number of texts in the corpus by the number of texts in the corpus containing the word, and then taking the logarithm of the quotient.

[0021] After calculating TF and IDF, multiplying the two values gives the TF-IDF value of the word. The larger the TF-IDF value, the higher the importance of the word to the text. Taking the top several words with the highest TF-IDF values in the text gives the keywords of the text.

[0022] According to exemplary embodiments, the short text similarity measure of the present disclosure can be determined by generating term frequency vectors for each of the two short texts and calculating the cosine similarity of the two term frequency vectors. According to other exemplary embodiments, the short text similarity measure of the present disclosure can be determined by generating term frequency vectors for each of the two short texts and calculating other similarity measures of the two term frequency vectors, such as including but not limited to jaccard similarity, Sorensen similarity coefficient, Levenshtein distance, and Hamming distance, etc.

[0023] Figure 1A schematic diagram of a short text similarity determination system 100 is shown according to an aspect of the present disclosure. As shown in FIG. 1, two or more short texts (e.g., short text 1 and short text 2) can be input into a word segmentation unit 102 for word segmentation respectively to obtain word sequences. The word segmentation can utilize various existing or future techniques. According to an exemplary embodiment, a language-specific word segmentation tool can be used. For example, a third-party word segmentation package such as jieba can be utilized, and other Chinese word segmentation packages such as THULAC, pkuseg, Hanlp, etc. can also be used. Figure 1 As shown in FIG. 1, two or more short texts (e.g., short text 1 and short text 2) can be input into a word segmentation unit 102 for word segmentation respectively to obtain word sequences. The word segmentation can utilize various existing or future techniques. According to an exemplary embodiment, a language-specific word segmentation tool can be used. For example, a third-party word segmentation package such as jieba can be utilized, and other Chinese word segmentation packages such as THULAC, pkuseg, Hanlp, etc. can also be used.

[0024] The word-segmented short texts (i.e., the word sequences of the short texts) can include different number of words. For example, short text 1 can be segmented into a word sequence of "word 1, word 2, …, word N", while short text 2 can be segmented into a word sequence of "word 1, word 2, …, word M", where N can not equal to M. In this case, padding or cutting can be employed to make the two equal in length. In addition, as can be appreciated, the expression such as "word n" is merely to represent that the word is the nth word in the short text, and the "word n" in different short texts need not be the same word.

[0025] The word-segmented short texts can be input into an optimized term frequency calculation unit 104. The optimized term frequency calculation unit 104 can calculate the term frequency of each word in each word-segmented short text. According to an exemplary embodiment, the term frequency in the present disclosure can be based on an optimized TF-IDF. That is, the optimized TF-IDF value of each word can be calculated, and the optimized TF-IDF values can be arranged in an order corresponding to the word sequence of each word in the short text to form a vector of optimized TF-IDF values as a term frequency vector corresponding to the short text. The calculation method of the optimized TF-IDF value is further described below.

[0026] The term frequency vector 1 and term frequency vector 2 corresponding to short text 1 and short text 2 respectively can be input into a similarity unit 106. The similarity unit 106 can determine the similarity of the term frequency vectors. According to an exemplary embodiment, the similarity of the term frequency vectors can be measured by the cosine distance of the vectors. Finally, the similarity unit 106 can provide the similarity measure of the term frequency vectors as the similarity measure of the short texts.

[0027] The short text similarity measure according to the present disclosure can be applied to various application scenarios, such as including but not limited to transaction commonly used shipping address aggregation, etc.

[0028] Although the above describes a method of calculating the similarity of two short texts at a time, the present disclosure is not limited thereto, but can cover embodiments of comparing the similarity of more short texts at a time, as long as it is based on the technology disclosed herein in essence. Figure 1

[0029] Figure 2 ​A schematic diagram of a term frequency determination apparatus 200 is shown in accordance with an aspect of the present disclosure. The term frequency determination apparatus 200 can include or constitute the above-described term frequency calculation unit 104. Figure 1 The term frequency calculation unit 104 is described.

[0030] According to exemplary embodiments, the term frequency determination apparatus 200 can include, but is not limited to, for example, a term frequency calculation unit 202, an inverse document frequency calculation unit 204, a term frequency weighting unit 206, and a term frequency vector generation unit 208, etc. Although the above-mentioned units are described herein as separate functional modules, one of ordinary skill in the art can know that the functions described in connection with each unit can be implemented by various techniques, for example, by software, firmware, general-purpose hardware, special-purpose hardware, etc. Furthermore, each unit need not be implemented by a separate software or hardware, but can be implemented, for example, by a general-purpose processor and a memory. The division of functions is merely for the convenience of one of ordinary skill in the art to understand, and does not constitute any limitation on the implementation of the present disclosure.

[0031] According to exemplary embodiments, the term frequency calculation unit 202 can obtain the segmented short texts 230 from the text library 210 to calculate the raw term frequency of individual terms. For example, the raw term frequency can be calculated as:

[0032]

[0033] According to exemplary embodiments, the inverse document frequency calculation unit 204 can calculate the inverse document frequency of the term in the corpus. For example, generally, the inverse document frequency can be calculated as:

[0034]

[0035] where n is the total number of texts in the corpus 220 obtained based on the text library 210, and m is the number of texts containing the term in the corpus. The number of texts containing the term is added by 1 to avoid the case that the denominator is 0.

[0036] The term frequency based on TF-IDF can have a strong dependence on the size of the corpus and the length of the text. For example, for a large corpus, the short text vector is very sparse, and the appearance of low-frequency terms seriously interferes with the similarity measure. On the other hand, for a small corpus, the IDF value fluctuates greatly, resulting in an unclear similarity.

[0037] Thus, the optimized IDF can be calculated. According to at least some exemplary embodiments, calculating the optimized IDF can include introducing a penalty, so that when the total number of texts in the corpus is small, the IDF is given a larger penalty to reduce the fluctuation of the IDF value; and when the total number of texts in the corpus is large, the penalty is reduced, and the penalty can gradually decrease as the total number of texts in the corpus gradually increases, ensuring that the weight coefficients of high-frequency terms and low-frequency terms are more stable.

[0038] According to an exemplary embodiment, the penalty may include a non-negative smoothing factor δ m When calculating the IDF of a specific word, the smoothing factor δ m It can be used to adjust the number of texts in the corpus that contain the word.

[0039] For example, when the total number of texts in the corpus is small (e.g., less than or equal to the first threshold), the smoothing factor δ m The number of texts in the corpus that contain the word is affected by the smoothing factor δ. m The IDF is significantly increased by adjustment, thereby reducing the jitter of the IDF value.

[0040] For example, according to some exemplary embodiments, when the total number of texts in the corpus is small (eg, less than or equal to a first threshold), the smoothing factor δ m According to some other exemplary embodiments, when the total number of texts in the corpus is small (for example, less than or equal to the first threshold), the smoothing factor δ m The change value can be taken as a value that decreases significantly as the total amount of text increases.

[0041] On the other hand, when the total number of texts in the corpus is large (for example, greater than the first threshold), the smoothing factor δ m It can be a value that decreases slowly as the total number of texts in the corpus increases. The number of texts in the corpus that contain the word is affected by the smoothing factor δ. m According to some exemplary embodiments, when the total number of texts in the corpus increases to a certain extent (e.g., greater than a second threshold), the smoothing factor δ m You can directly take 0.

[0042] According to some exemplary embodiments, the smoothing factor δ m Can include values ​​that decrease exponentially with the total number of texts in the corpus.

[0043] According to some exemplary embodiments, the smoothing factor δ is used m Adjusting the number of texts in the corpus that contain the word may include adding the smoothing factor δ to the number of texts in the corpus that contain the word. m .

[0044] According to some exemplary embodiments, the smoothing factor δ is used m Adjusting the number of texts in the corpus that contain the word may include multiplying the number of texts in the corpus that contain the word by the smoothing factor δ m .

[0045] According to some example embodiments, the first threshold of the total number of corpus texts can comprise, for example, 5. However, the present disclosure is not limited to this preferred embodiment. The first threshold can be larger or smaller.

[0046] According to some example embodiments, the second threshold of the total number of corpus texts can comprise, for example, 10. However, the present disclosure is not limited to this preferred embodiment. The second threshold can be larger or smaller.

[0047] According to some example embodiments, the smoothing factor δ m may be calculated as follows:

[0048]

[0049] where n is the total number of texts in the corpus. Accordingly, the smoothed IDF is calculated as follows: m

[0050]

[0051] According to some other example embodiments, the smoothing factor δ m may be calculated as follows:

[0052] δ m = ae (5-n) , m, n > 0 (5)

[0053] where n is the total number of texts in the corpus, m is the number of texts in the corpus containing the word, and a is a constant greater than 0. Accordingly, the smoothed IDF is calculated as follows: m

[0054]

[0055] As can be seen, the optimized IDF can comprise giving a larger penalty to the IDF when the total number of texts in the corpus is small, so as to reduce the jitter of the IDF value; and reducing the penalty when the total number of texts in the corpus is large, and the penalty can gradually decrease as the total number of texts in the corpus gradually increases, so as to ensure that the weight coefficients of high-frequency words and low-frequency words are more stable.

[0056] The present disclosure provides a method for calculating short text similarity by combining word segmentation and improved inverse document frequency, which adjusts the inverse document frequency value according to the size and number of texts of the corpus, and smooths the weight coefficients of high-frequency words and low-frequency words, so that the obtained text vector focuses on both the difference of low-frequency words and the similarity of high-frequency words, and the overall similarity is more stable and more accurate.

[0057] Figure 3 ​​FIG3 is a flow chart of a short text similarity determination method 300 according to an aspect of the present disclosure. The short text similarity determination method 300 may include, for example, performing word segmentation on the short text at block 302 to obtain a word sequence.

[0058] According to exemplary embodiments, word segmentation can utilize various existing or future technologies. According to exemplary embodiments, a word segmentation tool suitable for a specific language can be used. For example, a third-party word segmentation package such as Jieba can be used, or other Chinese word segmentation toolkits such as THULAC, pkuseg, Hanlp, etc. can be used.

[0059] According to an exemplary embodiment, the short text similarity determination method 300 may further include acquiring a text library, and performing word segmentation on the short texts in the text library as described in block 302 , thereby obtaining a word segmentation list.

[0060] According to an exemplary embodiment, the short text similarity determination method 300 may optionally further include removing stop words based on the word segmentation list to construct a corpus. Stop words refer to specific characters or words that are automatically filtered out in information retrieval involving natural language to save storage space and improve search efficiency.

[0061] The short text similarity determination method 300 may further include determining the term frequency (TF) and the adjusted inverse document frequency (IDF*) of each word in the short text, for example, at block 304. The adjusted inverse document frequency (IDF*) may be based on, for example, the above combination. Figure 1 and / or Figure 2 The method is described, so that when the total number of texts in the corpus is small, a larger IDF penalty is given to reduce the jitter of the IDF value; and when the total number of texts in the corpus is large, the penalty is reduced, and the penalty can be gradually reduced as the total number of texts in the corpus gradually increases, thereby ensuring that the weight coefficients of high-frequency words and low-frequency words are more stable.

[0062] The short text similarity determination method 300 may further include determining an optimized TF-IDF value for each word in the short text as the weighted term frequency of the word based on the term frequency (TF) and the optimized inverse document frequency (IDF*), for example, at block 306. The optimized TF-IDF value may be the product of the term frequency (TF) and the optimized inverse document frequency (IDF*) of each word in the short text.

[0063] After determining the weighted term frequency of each word in the short text, the short text similarity determination method 300 may further include determining a term frequency vector for the short text, for example, at block 308. Determining the term frequency vector for the short text may include combining the optimized TF-IDF value of each word into a vector in an order corresponding to the word sequence of the short text to form a term frequency vector corresponding to the short text.

[0064] The short text similarity determination method 300 can further include, for example at block 310, comparing the similarity between two or more short texts based on their term frequency vectors. According to some example embodiments, the similarity between two or more short texts can be determined based on the cosine similarity of their respective term frequency vectors. According to some example embodiments, the similarity between two or more short texts can be determined based on other similarity measures of their respective term frequency vectors.

[0065] Figure 4 A block diagram of a short text similarity determination apparatus 400 is shown, according to an aspect of the present disclosure. The short text similarity determination apparatus 400 can include, for example, a module 402 for tokenizing short texts.

[0066] According to example embodiments, the module 402 for tokenizing short texts can utilize various existing or future techniques. According to example embodiments, language-specific tokenization tools can be used. For example, third-party tokenization packages such as jieba can be utilized, and other Chinese tokenization packages such as THULAC, pkuseg, Hanlp, etc. can also be used.

[0067] According to example embodiments, the short text similarity determination apparatus 400 can further include a module (not shown) for obtaining a corpus of texts. The module 402 for tokenizing short texts can tokenize short texts in the obtained corpus of texts, thereby obtaining a tokenization list.

[0068] According to example embodiments, the short text similarity determination apparatus 400 can optionally further include a module (not shown) for removing stop words based on the tokenization list to construct a corpus. Stop words refer to specific words or terms that are automatically filtered out in information retrieval involving natural languages, in order to save storage space and improve search efficiency.

[0069] The short text similarity determination apparatus 400 can further include, for example, a module 404 for determining the term frequency (TF) and adjusted inverse document frequency (IDF*) of each term in the short texts. The adjusted inverse document frequency (IDF*) can be determined based on, for example, the manner described above in connection with Figure 1 and / or Figure 2 The adjusted inverse document frequency (IDF*) can be determined based on, for example, the manner described above in connection with

[0070] The short text similarity determination apparatus 400 can further include a module 406 for determining an optimized TF-IDF value for each word in a short text as a weighted term frequency for the word based on a term frequency (TF) and an adjusted inverse document frequency (IDF*). The optimized TF-IDF value can be a product of a term frequency (TF) and an adjusted inverse document frequency (IDF*) for each word in a short text.

[0071] The short text similarity determination apparatus 400 can further include a module 408 for determining a term frequency vector for a short text after determining the weighted term frequencies for each word in the short text. The module 408 for determining a term frequency vector for a short text can include, for example, a module for constituting a vector of optimized TF-IDF values in an order corresponding to the term order of each word in the short text into a term frequency vector corresponding to the short text.

[0072] The short text similarity determination apparatus 400 can further include a module 410 for comparing the similarity between two or more short texts based on the term frequency vectors of the short texts, for example. According to some example embodiments, the similarity between two or more short texts can be determined based on the cosine similarity of their respective term frequency vectors. According to some example embodiments, the similarity between two or more short texts can be determined based on other similarity measures of their respective term frequency vectors.

[0073] The short text similarity determination apparatus 400 can be implemented in software, firmware, and / or hardware, or any combination thereof. For example, the various modules of the short text similarity determination apparatus 400 can be implemented in the form of processor-executable instructions stored on a computer-readable storage medium, such that when one or more processors of a computer read and execute the processor-executable instructions, the computer performs the functions of the various modules 402-410, etc. of the short text similarity determination apparatus 400. As another example, the short text similarity determination apparatus 400 can be implemented by a combination of a processor and a memory coupled to the processor and executing a program and / or instructions stored in the memory to implement the functions of the various modules 402-410, etc. of the short text similarity determination apparatus 400. As yet another example, the short text similarity determination apparatus 400 can be implemented in other various special-purpose or general-purpose firmware / hardware.

[0074] The short text similarity efficient calculation method and device combining word segmentation and improved inverse text frequency according to various embodiments of the above aspects of the present disclosure eliminates or mitigates the influence of corpus size and text length on the similarity measurement between short texts, improves the inverse text frequency calculation method, thereby adaptively adjusting the post-inverse text frequency value according to the corpus size and the length of the text, smoothing the weight coefficients of high-frequency words and low-frequency words, and making the obtained text vector focus on both the difference of low-frequency words and the similarity of high-frequency words. The improved short text similarity efficient calculation method and device of inverse text frequency are more smooth, stable and easy to explain.

[0075] The above merely provides exemplary specific embodiments of the present application. The protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art, and shall be covered within the protection scope of the present application.

[0076] The various illustrative logical blocks, modules, and circuits described in connection with the present disclosure can be implemented or performed with a general-purpose processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, which are designed to perform the functions described herein. The general-purpose processor can be a microprocessor, but in the alternative, the processor can be any commercially available processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0077] The steps of a method or algorithm described in connection with the present disclosure can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in any form of storage medium known in the art. Some examples of storage media that can be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM and so forth. A software module can comprise a single instruction, or many instructions, and can be distributed over several different code segments, across several programs, and across several storage media. A storage medium can be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor.

[0078] The methods disclosed herein comprise one or more steps or actions for accomplishing a described method. The method steps and / or actions can be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims.

[0079] The software can be implemented as a routine 910 that is stored and executed by the processing system 900. Such software can also be stored and transmitted electronically over one or more computer networks. The software can also be embodied in various forms such as a program, procedures, functions, modules, methods, threads of execution, and / or the like, which can be executed by the processing system 900. The software can be implemented in a machine-readable medium, such as a storage device, memory, etc., to be executed by such software. The software can cause the processing system 900 to perform one or more of the methods described herein.

[0080] In hardware implementations, the machine-readable medium can be part of the processing system separate from the processor. However, as would be appreciated by those skilled in the art, the machine-readable medium or any portion of it can be external to the processing system. As an example, the machine-readable medium can include a transmission line, a carrier wave modulated by data, and / or a computer product, all of which can be accessed by the processor through the bus interface. Alternatively or in addition, the machine-readable medium or any portion of it can be integrated into the processor, such as the cache and / or general register files can be.

[0081] The processing system can be configured as a general- purpose processing system with one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable media, all linked together with an external bus architecture. Alternatively, the processing system can be implemented with an ASIC (Application Specific Integrated Circuit) with the processor, the bus interface, the user interface in the access terminal scenario, supporting circuitry, and at least a portion of the machine-readable media integrated into a single chip, or with one or more FPGAs (Field Programmable Gate Arrays), PLDs (Programmable Logic Devices), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuitry, or any combination of

[0082] The machine-readable media can comprise a number of software modules. The software modules include instructions that, when executed by an apparatus such as a processor, cause the processing system to perform various functions. The software modules can include a transmission module and a receiving module. Each software module can reside in a single storage device or be distributed across multiple storage devices. By way of example, a software module can be loaded into RAM from a hard drive when a triggering event occurs. During execution of the software module, the processor can load some of the instructions into cache to increase access speed. One or more cache lines can then be loaded into a general register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module.

[0083] If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Thus, in some aspects computer readable medium can comprise non-transitory computer readable medium (e.g., tangible media). In addition, for other aspects computer readable medium can comprise transitory computer readable medium (e.g., a signal). Combinations of the above should also be included within the scope of computer readable media.

[0084] Thus, certain aspects can comprise a computer program product for performing the operations presented herein. For example, such computer program product can comprise a computer readable medium having instructions stored thereon (and / or encoded therein) executable by one or more processors to perform the operations described herein. In certain aspects, the computer program product can include packaging material.

[0085] It will be appreciated that the claims are not limited to the precise configurations and components illustrated above. Various modifications, changes, and variations can be made in the method and apparatus described above without departing from the scope of the claims.​

Claims

1. A method for determining short text similarity, comprising: Segment the short texts in the corpus to obtain the corresponding word sequences; determining a penalty based on the total number of short texts in the corpus, wherein the penalty decreases as the total number of short texts in the corpus increases; Determining a word frequency and an adjusted inverse document frequency for each word in the word sequence, wherein the adjusted inverse document frequency is calculated based on the penalty, such that when the total number of texts in the corpus is small, a larger penalty is given to the inverse document frequency to reduce jitter in the inverse document frequency value; and when the total number of texts in the corpus is large, the penalty is reduced to make the weight coefficients of high-frequency words and low-frequency words more stable; weighting the term frequency of each term using the adjusted inverse document frequency; Combining the weighted word frequencies of each word in the word sequence to determine a word frequency vector for the short text; as well as The similarity between the short text and other short texts is determined based on the word frequency vector.

2. The method according to claim 1, wherein Determining the adjusted inverse document frequency for each term includes: determining the total number of texts in the corpus; determining the number of texts in the corpus that contain the word; adjusting the number of texts in the corpus containing the word based on the penalty, such that when the total number of short texts in the corpus is less than a first threshold, the number of texts in the corpus containing the word is exponentially increased; and The adjusted inverse document frequency is determined based on the total number of texts and the adjusted number of texts in the corpus that contain the word.

3. The method according to claim 2, wherein: Adjusting the number of texts in the corpus containing the word based on the penalty further enables the number of texts in the corpus containing the word to not be increased when the total number of short texts in the corpus is greater than a second threshold.

4. The method according to claim 2, wherein: The penalty includes an exponential smoothing factor, and the adjusting includes adding the exponential smoothing factor to the number of documents in the corpus that contain the word.

5. The method according to claim 1, wherein Weighting the term frequency of each term with the adjusted inverse document frequency includes multiplying the term frequency by the adjusted inverse document frequency calculated based on the penalty.

6. The method of claim 1, further comprising: If the word sequence obtained by segmenting the short text is different in length from the word sequence of the other short text, the word sequence of the short text or the word sequence of the other short text is padded or cut to make the two have the same length.

7. The method according to claim 1, wherein determining the similarity between the short text and other short texts based on the word frequency vector comprises: Calculate the cosine distance between the word frequency vector of the short text and the word frequency vector of the other short texts.

8. A short text similarity determination device, comprising: Memory; as well as a processor coupled to the memory, the processor configured to: Segment the short texts in the corpus to obtain the corresponding word sequences; determining a penalty based on the total number of short texts in the corpus, wherein the penalty decreases as the total number of short texts in the corpus increases; Determining a word frequency and an adjusted inverse document frequency for each word in the word sequence, wherein the adjusted inverse document frequency is calculated based on the penalty, such that when the total number of texts in the corpus is small, a larger penalty is given to the inverse document frequency to reduce jitter in the inverse document frequency value; and when the total number of texts in the corpus is large, the penalty is reduced to make the weight coefficients of high-frequency words and low-frequency words more stable; weighting the term frequency of each term using the adjusted inverse document frequency; Combining the weighted word frequencies of each word in the word sequence to determine a word frequency vector for the short text; as well as The similarity between the short text and other short texts is determined based on the word frequency vector.

9. The short text similarity determination device according to claim 8, wherein: The processor being configured to determine the adjusted inverse document frequency for each term includes the processor being configured to: determining the total number of texts in the corpus; determining the number of texts in the corpus that contain the word; adjusting the number of texts in the corpus containing the word based on the penalty, such that when the total number of short texts in the corpus is less than a first threshold, the number of texts in the corpus containing the word is exponentially increased; as well as The adjusted inverse document frequency is determined based on the total number of texts and the adjusted number of texts in the corpus that contain the word.

10. The short text similarity determination device according to claim 9, wherein: The processor being configured to adjust the number of texts in the corpus containing the word based on the penalty further includes the processor being configured such that when the total number of short texts in the corpus is greater than a second threshold, the number of texts in the corpus containing the word is not increased.

11. The short text similarity determination device according to claim 9, wherein: The penalty includes an exponential smoothing factor, and the adjusting includes adding the exponential smoothing factor to the number of documents in the corpus that contain the word.

12. The short text similarity determination device according to claim 8, wherein: The processor being configured to weight the term frequency of each term with the adjusted inverse document frequency includes the processor being configured to multiply the term frequency by the adjusted inverse document frequency calculated based on the penalty.

13. The short text similarity determination device according to claim 8, wherein the processor is further configured to: If the word sequence obtained by segmenting the short text is different in length from the word sequence of the other short text, the word sequence of the short text or the word sequence of the other short text is padded or cut to make the two have the same length.

14. The short text similarity determination device according to claim 8, wherein: The processor is configured to determine the similarity between the short text and other short texts based on the word frequency vector, including the processor being configured to: Calculate the cosine distance between the word frequency vector of the short text and the word frequency vector of the other short texts.

15. A short text similarity determination device, comprising: A module for tokenizing short texts in a corpus to obtain corresponding word sequences; module for determining a penalty based on a total number of short texts in the corpus, wherein the penalty decreases as the total number of short texts in the corpus increases; a module for determining a word frequency and an adjusted inverse document frequency for each word in the word sequence, wherein the adjusted inverse document frequency is calculated based on the penalty to give the inverse document frequency a larger penalty when the total number of texts in the corpus is small to reduce jitter in the inverse document frequency value; and to reduce the penalty when the total number of texts in the corpus is large to make the weight coefficients of high-frequency words and low-frequency words more stable; module for weighting the term frequency of each term by the adjusted inverse document frequency; A module for combining the weighted word frequencies of each word in the word sequence to determine a word frequency vector for the short text; as well as A module for determining similarity between the short text and other short texts based on the word frequency vector.

Images