New word discovery method, device, electronic device, and medium

Abstract

The present invention relates to data processing technology and discloses a new word discovery method, comprising: performing sentence segmentation, word segmentation and stop word removal processing on an original text set to obtain an original word set, merging a first preset number of consecutive words according to the order of the words in the original word set to generate a candidate word set, performing probability calculation on the candidate word set using a preset sequential n-gram language model to obtain probability values ​​of the candidate words and sort them, screening a second preset number of candidate words to obtain a screened word set, adding the probabilities of the same candidate words in the screened word set and sorting the candidate words, and selecting candidate words with probability values ​​greater than a preset threshold as new words. The present invention also relates to blockchain technology, and the new words can be stored in blockchain nodes. The present invention also discloses a new word discovery device, an electronic device and a computer-readable storage medium. The present invention can improve the efficiency and comprehensiveness of new word discovery.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a new word discovery method, device, electronic device and computer-readable storage medium. Background Art

[0002] The rapid development and popularization of informatization, electronicization and networking have brought about an explosive growth of information. A large number of new words that do not exist in traditional dictionaries are constantly emerging. With the rapid growth of various new words, in order to continuously improve the convenience of human-computer interaction, it is obviously necessary to continuously track new words and update the existing vocabulary.

[0003] Due to the huge scale of the Internet corpus, the method of manually screening and marking new words obviously cannot meet the requirements of practical application. The discovery of high-frequency character strings by statistical methods such as the information entropy, mutual information entropy, and word frequency of adjacent words is likely to miss possible new words and is not efficient enough in discovering new words. Therefore, it is necessary to propose a new word discovery method with more comprehensive screening. Summary of the Invention

[0004] The present invention provides a new word discovery method, device, electronic device and computer-readable storage medium, the main purpose of which is to solve the problem of incomplete new word discovery in the prior art.

[0005] To achieve the above-mentioned purpose, the present invention provides a new word discovery method, comprising:

[0006] Obtaining an original text set, performing sentence segmentation processing on the original text set to obtain an original sentence set, and performing word segmentation and stop word removal processing on the original sentence set to obtain an original word set;

[0007] According to the order of the words in the original word set, a first preset number of consecutive words in the original word set are merged to generate a candidate word set;

[0008] Using a preset sequential n-gram language model to perform probability calculation on each candidate word in the candidate word set to obtain a probability value for each candidate word in the candidate word set;

[0009] Filtering a second preset number of candidate words according to the ranking of the probability values ​​to obtain a filtered word set;

[0010] The probabilities of the same candidate words in the filtered word set are added, all the candidate words in the filtered word set are sorted according to the added probability values, and the candidate words with probability values ​​greater than a preset threshold are selected as new words.

[0011] Optionally, the sentence segmentation processing of the original text set to obtain an original sentence set includes:

[0012] Identifying a terminal punctuation mark in each original text in the original text set, and obtaining a position of the terminal punctuation mark;

[0013] The original text is segmented according to the position of the terminal punctuation mark to obtain original sentences, and all the original sentences are aggregated to obtain an original sentence set.

[0014] Optionally, the step of merging a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set includes:

[0015] Select one of the words from the original word set in turn through a traversal operation;

[0016] A first preset number of words following the currently selected word are sequentially obtained from the original word set, and a merging process is performed on the currently selected word and the first preset number of words to generate a candidate word set.

[0017] Optionally, the using a preset sequential n-gram language model to perform probability calculation on each candidate word in the candidate word set to obtain a probability value of each candidate word in the candidate word set includes:

[0018] Replacing candidate words in the candidate word set with a preset domain word dictionary to generate training samples;

[0019] A probability calculation is performed on the training samples using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set.

[0020] Optionally, the performing probability calculation on the training sample using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set includes:

[0021] The candidate word s is calculated using the following probability calculation formula k The probability value is:

[0022]

[0023]

[0024]

[0025] Among them, P(i) is the probability value, entity is the candidate word to be replaced, and k represents the word S k The position in the training sample, i represents the position of the entity in the training sample, n represents the number of candidate words after or before the entity, p pre (s k ) is the word Sk Calculate the forward probability relative to the entity, p suf (s k ) is the word S k Calculate the backward probability relative to entity.

[0026] Optionally, screening a second preset number of candidate words according to the ranking of the probability values ​​includes:

[0027] Sorting the candidate words according to the probability value of each candidate word in the candidate word set to generate a sorting queue;

[0028] A second preset number of candidate words having probability values ​​greater than a preset screening threshold are selected from the sorting queue to obtain a screening word set.

[0029] Optionally, the performing word segmentation and stop word removal processing on the original sentence set to obtain the original word set includes:

[0030] Divide the original sentence set from left to right according to a preset word segmentation rule to obtain a word segmentation set;

[0031] Performing stop word recognition processing on the word segmentation set to identify the stop words in the word segmentation set;

[0032] The stop words are deleted to obtain the original word set.

[0033] In order to solve the above problems, the present invention further provides a new word discovery device, comprising:

[0034] An original word set generation module is used to obtain an original text set, perform sentence segmentation processing on the original text set to obtain an original sentence set, and perform word segmentation and stop word removal processing on the original sentence set to obtain an original word set;

[0035] a merging module, configured to merge a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set;

[0036] A probability calculation module, configured to perform probability calculation on each candidate word in the candidate word set using a preset sequential n-gram language model to obtain a probability value for each candidate word in the candidate word set;

[0037] a screening module, configured to screen a second preset number of candidate words according to the ranking of the probability values ​​to obtain a screened word set;

[0038] The new word selection module is used to add the probabilities of the same candidate words in the filtered word set, sort all the candidate words in the filtered word set according to the added probability values, and select the candidate words with probability values ​​greater than a preset threshold as new words.

[0039] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0040] at least one processor; and,

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores computer program instructions that can be executed by the at least one processor. The computer program instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned new word discovery method.

[0043] In order to solve the above problem, the present invention further provides a computer-readable storage medium storing a computer program, which implements the above new word discovery method when executed by a processor.

[0044] The embodiment of the present invention performs sentence segmentation processing on the acquired original text set, which reduces the data dimension of the calculation and improves the calculation efficiency. The word segmentation and stop word removal processing reduces the frequency of invalid words and improves the discovery efficiency. Furthermore, a first preset number of consecutive words are merged and processed in the original word set to generate a candidate word set for new words, thereby improving the comprehensiveness of new word discovery. A preset sequential n-gram language model is used to perform probability calculation on each candidate word in the candidate word set, thereby improving the accuracy of the new word discovery task. Therefore, the new word discovery method, device, electronic device and computer-readable storage medium proposed in the present invention can improve the efficiency and comprehensiveness of new word discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a process for discovering new words according to an embodiment of the present invention;

[0046] Figure 2 for Figure 1 A flowchart of one of the steps in the new word discovery method shown;

[0047] Figure 3 for Figure 1 A flowchart of another step in the new word discovery method shown;

[0048] Figure 4 for Figure 1 A flowchart of another step in the new word discovery method shown;

[0049] Figure 5 A schematic diagram of modules of a new word discovery device provided by an embodiment of the present invention;

[0050] Figure 6A schematic diagram of the internal structure of an electronic device for implementing a new word discovery method provided by an embodiment of the present invention.

[0051] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] Embodiments of the present invention provide a new word discovery method. The method may be performed by at least one of a server, a terminal, or other electronic device capable of executing the method provided by the embodiments of the present application. In other words, the method may be performed by software or hardware installed on a terminal or server device, where the software may be a blockchain platform. The server may include, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0054] Reference Figure 1 FIG. 1 is a flow chart of a new word discovery method provided by an embodiment of the present invention. In this embodiment, the new word discovery method includes:

[0055] S1. Obtain an original text set, perform sentence segmentation processing on the original text set to obtain an original sentence set, and perform word segmentation and stop word removal on the original sentence set to obtain an original word set.

[0056] In the embodiment of the present invention, the original text set may be corpus data on the Internet. In the embodiment of the present invention, a Python statement with a data acquisition function is used to acquire the original text set from the Internet.

[0057] Specifically, the sentence segmentation processing is performed on the original text set to obtain the original sentence set, including:

[0058] Identifying a terminal punctuation mark in each original text in the original text set, and obtaining a position of the terminal punctuation mark;

[0059] The original text is segmented according to the position of the terminal punctuation mark to obtain original sentences, and all the original sentences are aggregated to obtain an original sentence set.

[0060] The terminal punctuation marks include, but are not limited to, periods, exclamation marks, question marks, ellipsis marks, quotation marks, brackets, etc.

[0061] For example, if the original text is aaa《ccc》ddd, it is processed into sentences, and the original sentences include "aaa", "ccc" and "ddd"; if the original text is aaa《ccc》ddd,ffff, it is processed into sentences, and the original sentence set includes "aaa", "ccc", "ddd" and "ffff".

[0062] The embodiment of the present invention performs sentence segmentation processing on the original text set, and can divide the text set into a set of multiple sentences, which facilitates subsequent operations and improves efficiency.

[0063] Furthermore, the original sentence set is segmented and stop word removed to obtain an original word set, including:

[0064] Divide the original sentence set from left to right according to a preset word segmentation rule to obtain a word segmentation set;

[0065] Performing stop word recognition processing on the word segmentation set to identify the stop words in the word segmentation set;

[0066] The stop words are deleted to obtain the original word set.

[0067] The stop words include but are not limited to auxiliary words, conjunctions and prepositions, and an original word set is obtained.

[0068] For example, the original sentence set is "My calf hurts a little bit". After the embodiment of the present invention performs word segmentation and removes stop words on the original sentence set, the original word set can be obtained as [I], [leg], [stomach], [a little bit], [hurt].

[0069] S2. According to the order of the words in the original word set, merge a first preset number of consecutive words in the original word set to generate a candidate word set.

[0070] See Figure 2 As shown, in the embodiment of the present invention, the step of merging a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set includes:

[0071] S21, selecting one word from the original word set in sequence through a traversal operation;

[0072] S22. Obtain a first preset number of words following the currently selected word in order from the original word set, perform merging processing on the currently selected word and the first preset number of words, and generate a candidate word set.

[0073] In an embodiment of the present invention, the first preset number may be 2, i.e., the embodiment of the present invention merges the currently selected word with the word following the currently selected word to generate a candidate word. For example, the original word set is [I], [leg], [stomach], [a little], [pain], and the candidate word set obtained by merging the original word set is [my leg], [calf], [stomach a little], [pain a little].

[0074] In another embodiment of the present invention, when performing the merging process, the following principles are followed: numbers and Chinese characters are not merged, letters and Chinese characters are not merged, etc.

[0075] S3. Calculate the probability of each candidate word in the candidate word set using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set.

[0076] See Figure 3 As shown, in the embodiment of the present invention, the probability calculation of each candidate word in the candidate word set using the preset sequential n-gram language model to obtain the probability value of each candidate word in the candidate word set includes:

[0077] S31, using a preset domain word dictionary to replace the candidate words in the candidate word set to generate training samples;

[0078] S32: Perform probability calculation on the training samples using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set.

[0079] The domain word dictionary contains known nouns in a preset domain.

[0080] Specifically, the embodiment of the present invention performs a traversal match between the candidate words in the candidate word set and the domain word dictionary. When a matching word is found in the domain word dictionary, the corresponding candidate word is replaced to generate a training sample. Preferably, if multiple matching words are found in the domain word dictionary, one of the corresponding candidate words is replaced each time to generate a corresponding training sample.

[0081] For example, the candidate word is "My legs hurt a little bit", and "leg" and "pain" in the candidate word are in the preset domain word dictionary. "Leg" and "pain" can be replaced with entity to generate corresponding training samples "My entity hurts a little bit" and training samples "My legs are a little entity".

[0082] Furthermore, the method of performing probability calculation on the training samples using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set includes:

[0083] The candidate word s is calculated using the following probability calculation formulak The probability value is:

[0084]

[0085]

[0086]

[0087] Among them, P(i) is the probability value, entity is the candidate word to be replaced, and k represents the word S k The position in the training sample, i represents the position of the entity in the training sample, n represents the number of candidate words after or before the entity, p pre (s k ) is the word S k Calculate the forward probability relative to the entity, p suf (s k ) is the word S k Calculate the backward probability relative to entity.

[0088] S4. Filter a second preset number of candidate words according to the ranking of the probability values ​​to obtain a filtered word set.

[0089] In an embodiment of the present invention, the second preset number may be set to 2. In an embodiment of the present invention, screening the second preset number of candidate words according to the sorting of the probability values ​​includes:

[0090] Sorting the candidate words according to the probability value of each candidate word in the candidate word set to generate a sorting queue;

[0091] A second preset number of candidate words having probability values ​​greater than a preset screening threshold are selected from the sorting queue to obtain a screening word set.

[0092] S5. Add the probabilities of the same candidate words in the filtered word set, sort all the candidate words in the filtered word set according to the added probability values, and select the candidate words with probability values ​​greater than a preset threshold as new words.

[0093] See Figure 4 As shown, in the embodiment of the present invention, the S5 includes:

[0094] S51, performing a summation process on the probabilities of the same candidate words in the screening word set, and sorting all the candidate words in the screening word set according to the probability values ​​after the summation process to obtain a sorting table;

[0095] S52: Determine the difference between the probability value in the ranking table and a preset threshold value;

[0096] When the probability value is less than or equal to the preset threshold, executing S53, the candidate word corresponding to the probability value is determined to be not a new word;

[0097] When the probability value is greater than the preset threshold, S54 is executed and the candidate word corresponding to the probability value is determined to be a new word.

[0098] For example, the probabilities that the candidate words in the screening word set are "calf" are summed to obtain a total probability value. When the total probability value is greater than a preset threshold, it is determined that "calf" is a new word.

[0099] In one embodiment of the present invention, the new word can be stored in a blockchain node.

[0100] The embodiment of the present invention performs sentence segmentation processing on the acquired original text set, which reduces the data dimension of the calculation and improves the calculation efficiency. The word segmentation and stop word removal processing reduces the frequency of invalid words and improves the discovery efficiency. Furthermore, a first preset number of consecutive words are merged and processed in the original word set to generate a candidate word set for new words, thereby improving the comprehensiveness of new word discovery. The preset sequential n-gram language model is used to perform probability calculation on each candidate word in the candidate word set, thereby improving the accuracy of the new word discovery task. Therefore, the present invention can improve the efficiency and comprehensiveness of new word discovery.

[0101] like Figure 5 , which is a module diagram of a new word discovery device provided by an embodiment of the present invention.

[0102] The new word discovery device 100 of the present invention can be installed in an electronic device. Depending on the functionality implemented, the new word discovery device 100 may include an original word set generation module 101, a merging module 102, a probability calculation module 103, a screening module 104, and a new word selection module 105. A module, also referred to as a unit, is a series of computer program segments that can be executed by an electronic device processor and perform a fixed function, and is stored in the memory of the electronic device.

[0103] In this embodiment, the functions of each module / unit are as follows:

[0104] The original word set generation module 101 is used to obtain an original text set, perform sentence segmentation processing on the original text set to obtain an original sentence set, and perform word segmentation and stop word removal processing on the original sentence set to obtain an original word set;

[0105] The merging module 102 is configured to merge a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set;

[0106] The probability calculation module 103 is used to perform probability calculation on each candidate word in the candidate word set using a preset sequential n-gram language model to obtain a probability value for each candidate word in the candidate word set;

[0107] The screening module 104 is configured to screen a second preset number of candidate words according to the ranking of the probability values ​​to obtain a screened word set;

[0108] The new word selection module 105 is configured to add the probabilities of identical candidate words in the filtered word set, sort all candidate words in the filtered word set according to the added probability values, and select candidate words with probability values ​​greater than a preset threshold as new words.

[0109] Specifically, when executed by a processor of an electronic device, each module in the new word discovery apparatus 100 can implement a new word discovery method comprising the following steps:

[0110] Step 1: The original word set generation module 101 obtains an original text set, performs sentence segmentation processing on the original text set to obtain an original sentence set, and performs word segmentation and stop word removal processing on the original sentence set to obtain an original word set.

[0111] In the embodiment of the present invention, the original text set may be corpus data on the Internet. The original word set generation module 101 in the embodiment of the present invention may obtain the original text set from the Internet using a Python statement with a data acquisition function.

[0112] Specifically, the original word set generation module 101 performs sentence segmentation processing on the original text set through the following operations to obtain an original sentence set:

[0113] Identifying a terminal punctuation mark in each original text in the original text set, and obtaining a position of the terminal punctuation mark;

[0114] The original text is segmented according to the position of the terminal punctuation mark to obtain original sentences, and all the original sentences are aggregated to obtain an original sentence set.

[0115] The terminal punctuation marks include, but are not limited to, periods, exclamation marks, question marks, ellipsis marks, quotation marks, brackets, etc.

[0116] For example, if the original text is aaa《ccc》ddd, it is processed into sentences, and the original sentences include "aaa", "ccc" and "ddd"; if the original text is aaa《ccc》ddd,ffff, it is processed into sentences, and the original sentence set includes "aaa", "ccc", "ddd" and "ffff".

[0117] The embodiment of the present invention utilizes the original word set generation module 101 to perform sentence segmentation processing on the original text set, and can divide the text set into a set of multiple sentences, which facilitates subsequent operations and improves efficiency.

[0118] Furthermore, the original word set generation module 101 performs word segmentation and stop word removal on the original sentence set through the following operations to obtain an original word set:

[0119] Divide the original sentence set from left to right according to a preset word segmentation rule to obtain a word segmentation set;

[0120] Performing stop word recognition processing on the word segmentation set to identify the stop words in the word segmentation set;

[0121] The stop words are deleted to obtain the original word set.

[0122] The stop words include but are not limited to auxiliary words, conjunctions and prepositions, and an original word set is obtained.

[0123] For example, the original sentence set is "My calf hurts a little bit". After the embodiment of the present invention performs word segmentation and removes stop words on the original sentence set, the original word set can be obtained as [I], [leg], [stomach], [a little bit], [hurt].

[0124] Step 2: The merging module 102 merges a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set.

[0125] In the embodiment of the present invention, the merging module 102 merges a first preset number of consecutive words in the original word set to generate a candidate word set by performing the following operations:

[0126] Select one of the words from the original word set in turn through a traversal operation;

[0127] A first preset number of words following the currently selected word are sequentially obtained from the original word set, and a merging process is performed on the currently selected word and the first preset number of words to generate a candidate word set.

[0128] In an embodiment of the present invention, the first preset number may be 2, i.e., the embodiment of the present invention merges the currently selected word with the word following the currently selected word to generate a candidate word. For example, the original word set is [I], [leg], [stomach], [a little], [pain], and the candidate word set obtained by merging the original word set is [my leg], [calf], [stomach a little], [pain a little].

[0129] In another embodiment of the present invention, when performing the merging process, the merging module 102 follows the following principles: numbers and Chinese characters are not merged, letters and Chinese characters are not merged, etc.

[0130] Step 3: The probability calculation module 103 uses a preset sequential n-gram language model to perform probability calculation on each candidate word in the candidate word set to obtain a probability value of each candidate word in the candidate word set.

[0131] In the embodiment of the present invention, the probability calculation module 103 performs probability calculation on each candidate word in the candidate word set by the following operations to obtain a probability value of each candidate word in the candidate word set:

[0132] Replacing candidate words in the candidate word set with a preset domain word dictionary to generate training samples;

[0133] A probability calculation is performed on the training samples using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set.

[0134] The domain word dictionary contains known nouns in a preset domain.

[0135] Specifically, the probability calculation module 103 of the embodiment of the present invention traverses and matches the candidate words in the candidate word set with the domain word dictionary. When a matching word is found in the domain word dictionary, the corresponding candidate word is replaced to generate a training sample. Preferably, if multiple matching words are found in the domain word dictionary, one of the corresponding candidate words is replaced each time to generate a corresponding training sample.

[0136] For example, the candidate word is "My legs hurt a little bit", and "leg" and "pain" in the candidate word are in the preset domain word dictionary. "Leg" and "pain" can be replaced with entity to generate corresponding training samples "My entity hurts a little bit" and training samples "My legs are a little entity".

[0137] Furthermore, the probability calculation module 103 calculates the candidate word s using the following probability calculation formula: k The probability value is:

[0138]

[0139]

[0140]

[0141] Among them, P(i) is the probability value, entity is the candidate word to be replaced, and k represents the word S kThe position in the training sample, i represents the position of the entity in the training sample, n represents the number of candidate words after or before the entity, p pre (s k ) is the word S k Calculate the forward probability relative to the entity, p suf (s k ) is the word S k Calculate the backward probability relative to entity.

[0142] Step 4: The screening module 104 screens a second preset number of candidate words according to the ranking of the probability values ​​to obtain a screened word set.

[0143] In an embodiment of the present invention, the second preset number may be set to 2. In an embodiment of the present invention, the screening module 104 screens the second preset number of candidate words according to the ranking of the probability values, including:

[0144] Sorting the candidate words according to the probability value of each candidate word in the candidate word set to generate a sorting queue;

[0145] A second preset number of candidate words having probability values ​​greater than a preset screening threshold are selected from the sorting queue to obtain a screening word set.

[0146] Step 5: The new word selection module 105 adds the probabilities of the same candidate words in the filtered word set, sorts all the candidate words in the filtered word set according to the added probability values, and selects the candidate words with probability values ​​greater than a preset threshold as new words.

[0147] Specifically, the new word selection module 105 determines the candidate word as the new word in the following operations:

[0148] performing a summation process on the probabilities of the same candidate words in the screening word set, and sorting all the candidate words in the screening word set according to the probability values ​​after the summation process to obtain a sorting table;

[0149] Determine the difference between the probability value in the ranking table and a preset threshold;

[0150] When the probability value is less than or equal to the preset threshold, the candidate word corresponding to the probability value is determined to be not a new word;

[0151] When the probability value is greater than the preset threshold, the candidate word corresponding to the probability value is determined to be a new word.

[0152] For example, the probabilities that the candidate word in the screening word set is "calf" are summed to obtain a total probability value. If the total probability value is greater than a preset threshold, it is determined that "calf" is a new word.

[0153] In one embodiment of the present invention, the new word can be stored in a blockchain node.

[0154] like Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing the new word discovery method of the present invention.

[0155] The electronic device 1 may include a processor 10 , a memory 11 , and a bus, and may further include a computer program stored in the memory 11 and executable on the processor 10 , such as a new word discovery program 12 .

[0156] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Furthermore, the memory 11 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed on the electronic device 1, such as the code of the new word discovery program 12, etc., but can also be used to temporarily store data that has been output or is to be output.

[0157] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines. It executes or executes programs or modules stored in the memory 11 (such as executing a new word discovery program) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0158] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0159] Figure 6 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 6 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0160] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering the various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0161] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0162] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0163] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0164] The new word discovery program 12 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve the following:

[0165] Obtaining an original text set, performing sentence segmentation processing on the original text set to obtain an original sentence set, and performing word segmentation and stop word removal processing on the original sentence set to obtain an original word set;

[0166] According to the order of the words in the original word set, a first preset number of consecutive words in the original word set are merged to generate a candidate word set;

[0167] Using a preset sequential n-gram language model to perform probability calculation on each candidate word in the candidate word set to obtain a probability value for each candidate word in the candidate word set;

[0168] Filtering a second preset number of candidate words according to the ranking of the probability values ​​to obtain a filtered word set;

[0169] The probabilities of the same candidate words in the filtered word set are added, all the candidate words in the filtered word set are sorted according to the added probability values, and the candidate words with probability values ​​greater than a preset threshold are selected as new words.

[0170] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0171] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0172] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0173] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0174] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0175] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0176] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a table in a claim should not be construed as limiting the claim.

[0177] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A new word discovery method, characterized in that: The method comprises: Obtaining an original text set, performing sentence segmentation processing on the original text set to obtain an original sentence set, and performing word segmentation and stop word removal on the original sentence set to obtain an original word set; According to the order of the words in the original word set, a first preset number of consecutive words in the original word set are merged to generate a candidate word set; Traversing and matching candidate words in the candidate word set in a preset domain word dictionary, obtaining one or more matching words found in the domain word dictionary, replacing one of the corresponding candidate words with the matching words in sequence, generating corresponding training samples, and performing probability calculation on the training samples using a preset sequential n-gram language model to obtain a probability value for each candidate word in the candidate word set; Filtering a second preset number of candidate words according to the ranking of the probability values ​​to obtain a filtered word set; The probabilities of the same candidate words in the filtered word set are added, all the candidate words in the filtered word set are sorted according to the added probability values, and the candidate words with probability values ​​greater than a preset threshold are selected as new words.

2. The new word discovery method according to claim 1, wherein: The sentence segmentation process is performed on the original text set to obtain an original sentence set, including: Identifying a terminal punctuation mark in each original text in the original text set, and obtaining a position of the terminal punctuation mark; The original text is segmented according to the position of the terminal punctuation mark to obtain original sentences, and all the original sentences are aggregated to obtain an original sentence set.

3. The new word discovery method according to claim 1, wherein: The step of merging a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set includes: Select one of the words from the original word set in turn through a traversal operation; A first preset number of words following the currently selected word are sequentially obtained from the original word set, and a merging process is performed on the currently selected word and the first preset number of words to generate a candidate word set.

4. The new word discovery method according to claim 1, wherein: The method of performing probability calculation on the training sample using a preset sequential n-gram language model to obtain a probability value of each candidate word in the candidate word set includes: The candidate word s is calculated using the following probability calculation formula k The probability value is: Among them, P(i) is the probability value, entity is the candidate word to be replaced, and k represents the word S k The position in the training sample, i represents the position of the entity in the training sample, n represents the number of candidate words after or before the entity, p pre (s k ) is the word S k Calculate the forward probability relative to the entity, p suf (s k ) is the word S k Calculate the backward probability relative to entity.

5. The new word discovery method according to claim 4, wherein: The step of screening a second preset number of candidate words according to the ranking of the probability values ​​includes: Sorting the candidate words according to the probability value of each candidate word in the candidate word set to generate a sorted queue; A second preset number of candidate words having probability values ​​greater than a preset screening threshold are selected from the sorting queue to obtain a screening word set.

6. The new word discovery method according to claim 1, wherein: The processing of segmenting and removing stop words on the original sentence set to obtain an original word set includes: Divide the original sentence set from left to right according to a preset word segmentation rule to obtain a word segmentation set; Performing stop word recognition processing on the word segmentation set to identify the stop words in the word segmentation set; The stop words are deleted to obtain the original word set.

7. A new word discovery device, characterized in that: The device comprises: An original word set generation module is used to obtain an original text set, perform sentence segmentation processing on the original text set to obtain an original sentence set, and perform word segmentation and stop word removal processing on the original sentence set to obtain an original word set; a merging module, configured to merge a first preset number of consecutive words in the original word set according to the order of the words in the original word set to generate a candidate word set; a probability calculation module, configured to traverse and match candidate words in the candidate word set in a preset domain word dictionary, obtain one or more matching words found in the domain word dictionary, replace one of the corresponding candidate words with the matching words in sequence, generate corresponding training samples, and perform probability calculation on the training samples using a preset sequential n-gram language model to obtain a probability value for each candidate word in the candidate word set; a screening module, configured to screen a second preset number of candidate words according to the ranking of the probability values ​​to obtain a screened word set; The new word selection module is used to add the probabilities of the same candidate words in the filtered word set, sort all the candidate words in the filtered word set according to the added probability values, and select the candidate words with probability values ​​greater than a preset threshold as new words.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor. The computer program instructions are executed by the at least one processor to enable the at least one processor to perform the new word discovery method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the new word discovery method according to any one of claims 1 to 6 is implemented.

Images