Method, device, electronic device and storage medium for building a vocabulary library

Abstract

The present application discloses a vocabulary construction method, device, electronic device and storage medium, belonging to the technical field of input method. The vocabulary construction method includes: processing each word segmentation in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are positive integers greater than 1; respectively screening out a tuple set including preset characters from each i-tuple set, and correspondingly obtaining a target i-tuple set; wherein the preset characters include characters input in at least two input modes; counting the frequency of each tuple in the target tuple set; wherein the target tuple set is a set in N i-tuple sets; determining the target tuple based on the frequency, and constructing a vocabulary based on the target tuple.

Description

Technical Field

[0001] The present application belongs to the field of input methods, and specifically relates to a vocabulary building method, device, electronic device and storage medium. Background Art

[0002] There are a lot of words in life that are mixed with Chinese, English and numbers. Currently, when using input methods, users often input mixed words by constantly switching keyboard types, which affects input efficiency.

[0003] For input method applications, how to construct a vocabulary that includes mixed words is an important research issue at present.

[0004] There are two main ways to construct a mixed word vocabulary: (1) directly use a third-party open source tool to segment a sentence containing English and numbers. However, this method usually directly treats the English and number fragments as a word respectively. However, a single English or number has no obvious semantic orientation. Therefore, the mixed word vocabulary constructed by the word method is unreasonable. (2) first manually construct a mixed word vocabulary and then use a third-party open source tool to segment the words. However, this method requires a lot of manpower and is inefficient. In addition, for words that are not maintained manually, the subsequent segmentation using open source tools will still have the defects of the first solution. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a vocabulary building method, device, electronic device and storage medium, which can solve the problems of unreasonable construction of mixed word vocabulary in input methods in the prior art, as well as the need for a lot of manpower and low efficiency.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a vocabulary library, the method comprising:

[0007] Process each word in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are positive integers greater than 1;

[0008] Filtering out a tuple set including preset characters from each i-tuple set respectively, and obtaining a corresponding target i-tuple set; wherein the preset characters include characters input in at least two input modes;

[0009] Counting the frequency of each tuple in the target tuple set; wherein the target tuple set is a set in N i-tuple sets;

[0010] A target tuple is determined based on the frequency, and a vocabulary is constructed based on the target tuple.

[0011] In a second aspect, an embodiment of the present application provides a vocabulary building device, the device comprising:

[0012] The first processing module is used to process each word segmentation in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are both positive integers greater than 1;

[0013] A screening module, used to screen out tuple sets including preset characters from each i-tuple set, and obtain corresponding target i-tuple sets respectively; wherein the preset characters include characters input in at least two input modes;

[0014] A statistical module, used for counting the frequency of each tuple in a target tuple set; wherein the target tuple set is a set in N i-tuple sets;

[0015] A construction module is used to determine a target tuple based on the frequency and to construct a vocabulary based on the target tuple.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0020] In an embodiment of the present application, each word segmentation in the first word segmentation result corresponding to the first corpus is processed to obtain N i-tuple sets, and then a tuple set including preset characters is screened out from each i-tuple set, and corresponding target i-tuple sets are obtained respectively. Since the preset characters include characters input in at least two input modes, the tuples in the target i-tuple set obtained in this way include characters input in at least two input modes. In this way, the vocabulary constructed according to the statistical frequency of each tuple in the target tuple set includes characters input in at least two input modes, ensuring that the preset characters with complete semantics will not be separated, and in the process of constructing the vocabulary, a large amount of manpower is not required, thereby improving the efficiency of vocabulary construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a vocabulary building method according to an exemplary embodiment;

[0022] Figure 2 is a schematic diagram showing a first word segmentation result obtained by performing word segmentation processing on a first corpus according to an exemplary embodiment;

[0023] Figure 3 is a schematic diagram of processing the first word segmentation result to obtain an i-tuple set according to an exemplary embodiment;

[0024] Figure 4 is a schematic diagram of screening an i-tuple set to obtain a target i-tuple set according to an exemplary embodiment;

[0025] Figure 5 is a schematic diagram showing a method of determining a first candidate tuple according to an exemplary embodiment;

[0026] Figure 6 is a schematic diagram of a second word segmentation result obtained by performing word segmentation on a second corpus using the word library construction method provided in an embodiment of the present application according to an exemplary embodiment;

[0027] Figure 7 is a structural schematic diagram of a vocabulary building device according to an exemplary embodiment;

[0028] Figure 8 is a structural block diagram of an electronic device according to an exemplary embodiment;

[0029] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] The embodiments of the present application provide a vocabulary construction method, device, electronic device and storage medium, which processes each word in the first word segmentation result corresponding to the first corpus to obtain an i-tuple set, and then selects a tuple set including preset characters from each i-tuple set, and obtains corresponding target i-tuple sets respectively. Since the preset characters include characters input in at least two input modes, the tuples in the target i-tuple set obtained in this way include characters input in at least two input modes. In this way, the vocabulary constructed according to the statistical frequency of each tuple in the target tuple set includes characters input in at least two input modes, ensuring that the preset characters with complete semantics will not be separated, and in the process of constructing the vocabulary, a large amount of manual work is not required, thereby improving the efficiency of vocabulary construction.

[0033] The following is a detailed description of the vocabulary building method provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0034] Figure 1 It is a flow chart of a vocabulary building method provided in an embodiment of the present application. The execution subject of the vocabulary building method may be a server. It should be noted that the above execution subject does not constitute a limitation on the embodiment of the present application.

[0035] like Figure 1 As shown, the vocabulary library construction method provided in the embodiment of the present application may include steps 110 to 140.

[0036] The following is a detailed description of the method for constructing a legal vocabulary library in the embodiment of the present application.

[0037] Step 110: Process each word segmentation in the first word segmentation result corresponding to the first corpus to obtain an i-tuple set.

[0038] The first corpus may be the acquired text information, or may be the text information obtained by performing phonetic-character translation based on the voice information, which is not limited here.

[0039] In one example, the first corpus may be a paragraph or several paragraphs, such as Figure 1 As shown, the first corpus can be "The weather is nice today", "Who is your idol", "AB is my idol", "Superstar AB debut" and "The answer to this question is C, right?"

[0040] It should be noted that "AB" in the above example is a name composed of numbers, Chinese and English. For example, it can be a name composed of English and Chinese, that is, A is an English letter and B is a Chinese character.

[0041] In some embodiments of the present application, the first word segmentation result may be a word segmentation result obtained after word segmentation of the first corpus.

[0042] Correspondingly, in some embodiments of the present application, before step 110, the above-mentioned word library construction method may further include:

[0043] The first corpus is segmented to obtain a first segmentation result.

[0044] In some embodiments of the present application, the word segmentation of the first corpus may be performed using an existing word segmentation tool, thereby improving the efficiency of the word segmentation of the first corpus.

[0045] In one example, continue to refer to Figure 1 ,right Figure 1 After the first corpus in is segmented, the first segmentation result is as follows Figure 1 As shown in the right figure.

[0046] The i-tuple set may be a set of tuples obtained after processing each word in the first word segmentation result, where i∈[1, k], k is a positive integer.

[0047] In some embodiments of the present application, in order to further obtain a reasonable input method vocabulary, step 110 may specifically include:

[0048] When i=1, each word in the first word segmentation result is formed into an independent tuple to obtain a unit group set;

[0049] When i>1, each word in the first word segmentation result is combined in the same way as i adjacent words are combined to obtain a multi-tuple set.

[0050] The unit group set may be a set consisting of independent tuples formed by forming each word in the first word segmentation result.

[0051] The multi-tuple set may be a set formed by combining each word in the first word segmentation result in a manner of combining i adjacent word segments.

[0052] In an example, taking the first corpus as "The weather is so nice today", after word segmentation processing of the first corpus, the first word segmentation result is "today" / "weather" / "really nice" / "yeah". When i=1, each word segmentation in the above "today" / "weather" / "really nice" / "yeah" is formed into a tuple separately, that is, the unit group set is obtained: (today), (weather), (really nice) and (yeah).

[0053] Continuing with the above example, when i=2, each word in the first word segmentation result "today" / "weather" / "really nice" / "ah" is combined with two adjacent words to form a 2-tuple set: (today, weather), (weather, really nice) and (really nice, ah).

[0054] Continuing with the parameter example, when i=3, the first word segmentation result "today" / "weather" / "really good" / "ah" is combined into a 3-tuple set by combining the three adjacent word segments: (today, weather, really good) and (weather, really good, ah). By analogy, a tuple set can be formed.

[0055] In another example, if Figure 2 As shown, the first corpus is Figure 1 Take the right picture in as an example, after getting the first word segmentation result ( Figure 2 As shown in the right figure of ), we can get the i-tuple set according to the first word segmentation result, and we can get Figure 3 The result is shown on the right.

[0056] It should be noted that for the sake of brevity, Figure 3 The unit group set is omitted in the right figure, but it does not mean that the unit group set is generated.

[0057] In an embodiment of the present application, when i=1, each word in the first word segmentation result can be formed into an independent tuple to obtain a single-unit group set; when i>1, each word in the first word segmentation result can be combined in the same manner as i adjacent words to obtain a multi-unit group set, thereby obtaining an accurate i-tuple set, and then based on the i-tuple set, a reasonable input method vocabulary can be obtained.

[0058] Step 120 , filter out tuple sets including preset characters from each i-tuple set, and obtain corresponding target i-tuple sets respectively.

[0059] The preset characters may be preset characters, and the preset characters here may be characters input in at least two input modes.

[0060] In one example, if the input method is mainly in Chinese, the preset characters here may be characters containing English letters and / or numbers. If the input method is mainly in English letters, the preset characters here may be characters containing Chinese and / or numbers. That is, the preset characters here may include characters input in at least two input modes.

[0061] The target i-tuple set may be a set formed by selecting tuples including preset characters from each i-tuple set.

[0062] In one example, tuples including preset characters may be selected from the unit group set to obtain a target unit group set. Tuples including preset characters may be selected from the 2-tuple set to obtain a target 2-tuple set. Tuples including preset characters may be selected from the 3-tuple set to obtain a target 3-tuple set. And so on, a target i-tuple set may be obtained.

[0063] In a specific example, if Figure 4 As shown, the input method is mainly Chinese, and the first corpus is Figure 1 Take the right picture as an example, Figure 3 After obtaining the i-tuple set shown in the right figure in , tuples including English letters and numbers can be filtered out from each i-tuple set to obtain the corresponding target i-tuple set.

[0064] Step 130: Count the frequency of each tuple in the target tuple set.

[0065] The target tuple set may be a set among N i-tuple sets, and specifically may be a tuple set among sets among i-tuple sets.

[0066] Step 140: Determine a target tuple based on the frequency, and build a vocabulary based on the target tuple.

[0067] The target tuple may be a tuple determined based on frequency.

[0068] In some embodiments of the present application, in order to further obtain an input method vocabulary that ensures that preset characters with complete semantics will not be separated, step 130 may specifically include:

[0069] The frequency of each tuple in the target q-tuple set appearing in the target q+j-tuple set is counted, where q is a positive integer greater than or equal to 2.

[0070] Correspondingly, step 140 may specifically include:

[0071] Filter out each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold, and obtain a first candidate tuple; wherein j is a positive integer greater than or equal to 1;

[0072] Based on the second candidate tuple, a vocabulary is constructed.

[0073] in,

[0074] The preset frequency threshold may be a preset threshold of the frequency of each tuple in the target q-tuple set appearing in the target q+j-tuple set. The threshold may be set according to user needs and is not limited here.

[0075] The first candidate tuples may be tuples in the screened target q-tuple set whose frequencies are greater than or equal to a preset frequency threshold.

[0076] In one example, if Figure 5 As shown, taking q=2, j=1 as an example, we get Figure 4 After obtaining the target tuple set shown in the right figure, the frequency of each tuple (A, B) / (star, A) / (yes, C) / (C, bar) in the target 2-tuple set in the target 3-tuple set (A, B, yes) / (super, star, A) / (star, A, B) / (A, B, first release)(answer, yes, C) / (yes, C, bar) is counted. When the frequency is greater than or equal to a preset frequency threshold (for example, the preset frequency threshold may be 2), the 2-tuple (A, B) with a frequency greater than or equal to the preset frequency threshold is filtered out to obtain the first candidate tuple.

[0077] In some embodiments of the present application, after obtaining the first candidate tuple, an input method vocabulary may be constructed based on the first candidate tuple. Specifically, each first candidate tuple may be taken as a whole to construct the input method vocabulary.

[0078] In an embodiment of the present application, the frequency of occurrence of each tuple in the target q-tuple set in the target q+j-tuple set is counted, and each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold is screened out to obtain a first candidate tuple; then, based on the first candidate tuple, an input method vocabulary is constructed. The input method vocabulary constructed in this way can subsequently ensure that preset characters with complete text semantics will not be separated, thereby obtaining a more reasonable input method vocabulary. In addition, in the process of constructing the input method vocabulary, no manual participation is required, thereby improving the construction efficiency of the input method vocabulary.

[0079] In some embodiments of the present application, the above is for the target tuple. In order to further obtain an input method vocabulary that ensures that the preset characters with complete semantics will not be separated, before determining the target tuple based on the frequency and constructing the input method vocabulary based on the target tuple, the vocabulary construction method involved in the above may also include:

[0080] For the target unit group set, remove the tuples without text meaning from the target unit group set to obtain the second candidate tuple;

[0081] Correspondingly, constructing the input method vocabulary based on the first candidate tuple may specifically include:

[0082] An input method vocabulary is constructed based on the first candidate tuple and the second candidate tuple.

[0083] The no-text meaning means no specific text semantics. For example, a single number or an English letter has no special semantics. Such a tuple can be called a tuple without text meaning.

[0084] The second candidate tuple may be a tuple obtained by removing tuples without textual meaning from the target unit group set and retaining the remaining unit groups.

[0085] In one example, continue to refer to Figure 4 After obtaining the target unit group set: (C) and (A), remove the tuples without textual meaning in the target unit group set, that is, remove a single number or an English letter in the target unit group set. Figure 4 Remove tuple (C) and tuple (A) from the set, and then retain the remaining tuples in the unit group set to obtain the first candidate tuple.

[0086] In an embodiment of the present application, for a target unit group set, tuples without textual meaning are removed from the target unit group set to obtain a second candidate tuple, and then an input method vocabulary is constructed based on the first candidate tuple and the second candidate tuple. The input method vocabulary constructed in this way can subsequently ensure that preset characters with complete textual semantics will not be separated, thereby obtaining a more reasonable input method vocabulary. In addition, in the process of constructing the input method vocabulary, no human participation is required, thereby improving the construction efficiency of the input method vocabulary.

[0087] In some embodiments of the present application, in order to prevent the user from constantly switching the input method keyboard when inputting characters in at least two different input modes, after step 140, the above-mentioned word library construction method may further include:

[0088] The language model is trained based on the input method vocabulary to obtain a trained target language model.

[0089] The target language model may be a trained language model obtained by training a language model based on a constructed input method vocabulary.

[0090] In some embodiments of the present application, specifically when using an input method vocabulary to train a language model, the input method vocabulary can be used to perform word segmentation on the corpus, and then the frequency of occurrence of each tuple in each i-tuple set is counted, and the probability of two word segments appearing in adjacent positions is calculated based on the frequency, and the loss function of the language model is determined based on the probability value. When the loss function is less than a preset threshold, it is determined that the language model training is completed.

[0091] In an embodiment of the present application, the speech model can be trained through the constructed input method vocabulary to obtain a target language model, so that the corpus to be segmented can be subsequently segmented based on the target language model, thereby obtaining a segmentation result in which the preset characters with complete semantics will not be separated.

[0092] In some embodiments of the present application, after the language model is trained based on the input method vocabulary to obtain the trained target language model, the vocabulary construction method involved above may further include:

[0093] The second corpus is segmented based on the target language model to obtain a second segmentation result.

[0094] The second corpus may be the corpus to be segmented using the target language model. The second corpus is the same as the first corpus, and may be the acquired text information or the text information obtained by performing phonetic-character translation based on the voice information, which is not limited here.

[0095] The second word segmentation result may be a word segmentation result obtained after performing word segmentation processing on the second corpus using the target language model.

[0096] In some embodiments of the present application, the second word segmentation result may include words composed of characters input in at least two input modes.

[0097] In one example, if Figure 6 As shown in FIG. 1 , after obtaining the target language model, the target language model can be used to train the second corpus (here the second corpus is still in Figure 1 Take the first corpus in the example as an example) and we will get the following result: Figure 6 The word segmentation result shown in the right figure (i.e., the second word segmentation result), in which "AB" (the "AB" word segmentation has characters entered in two input modes, namely, the character "A" entered in the English input mode, and the character "B" entered in the Chinese input mode) will not be segmented into "A" and "B".

[0098] In some embodiments of the present application, if you want to input "Q7" using the current input method keyboard (such as a 9-key keyboard), you must first switch to the English input mode, enter "Q", and then switch to the numeric input mode and enter "7". However, the target language model obtained in the embodiments of the present application is embedded in the input method, and the user can directly use the input method keyboard (9-key keyboard) to enter "774", and the candidate word "Q7" will appear directly (if you enter "774" on the existing 9-key input method keyboard, the candidate word "Q7" will not appear). In this way, using the vocabulary construction method provided in the embodiments of the present application, it can be achieved that if the user wants to input characters input in at least two input modes at the front end, there is no need to switch the input mode, which improves the user's input experience and fundamentally improves the quality of the language model.

[0099] In an embodiment of the present application, by performing word segmentation processing on the second corpus based on the target language model, the second word segmentation result, since the target language model is constructed based on the input method vocabulary, the second word segmentation result obtained after the word segmentation processing based on the target language model contains words composed of characters input in at least two input modes, thereby achieving the effect that two words with complete semantics will not be separated.

[0100] The lexicon construction method provided in the embodiment of the present application can be executed by a lexicon construction device. In the embodiment of the present application, the lexicon construction method executed by the lexicon construction device is taken as an example to illustrate the lexicon construction device provided in the embodiment of the present application.

[0101] Based on the same inventive concept as the above-mentioned word library construction method, the present application also provides a word library construction device. Figure 7 The vocabulary library construction device provided in the embodiment of the present application is described in detail.

[0102] Figure 7 The figure is a structural block diagram of a vocabulary building device according to an exemplary embodiment.

[0103] like Figure 7 As shown, the word library construction device 700 may include:

[0104] The first processing module 710 is used to process each word segmentation in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are both positive integers greater than 1;

[0105] A screening module 720 is used to screen out a tuple set including preset characters from each i-tuple set, and obtain a corresponding target i-tuple set respectively; wherein the preset characters include characters input in at least two input modes;

[0106] A statistical module 730 is used to count the frequency of each tuple in a target tuple set; wherein the target tuple set is a set in N i-tuple sets;

[0107] The construction module 740 is used to determine a target tuple based on the frequency and construct a vocabulary based on the target tuple.

[0108] In an embodiment of the present application, each word segmentation in the first word segmentation result corresponding to the first corpus is processed by a first processing module to obtain an i-tuple set, and then a tuple set including preset characters is screened out from each i-tuple set based on a screening module, and corresponding target i-tuple sets are obtained respectively. Since the preset characters include characters input in at least two input modes, the tuples in the target i-tuple set obtained in this way include characters input in at least two input modes. In this way, the input method vocabulary constructed according to the statistical frequency of each tuple in the target tuple set includes characters input in at least two input modes, thereby ensuring that the preset characters with complete semantics will not be separated, and in the process of constructing the vocabulary, a large amount of manpower is not required, thereby improving the efficiency of vocabulary construction.

[0109] In some embodiments of the present application, in order to further obtain a reasonable input method vocabulary, the first processing module 710 may specifically include:

[0110] A first processing unit is used for, when i=1, forming each word in the first word segmentation result into an independent tuple to obtain a unit group set;

[0111] The second processing unit is used for combining each word in the first word segmentation result in the manner of combining i adjacent word segmentations to obtain a multi-tuple set when i>1.

[0112] In some embodiments of the present application, in order to further obtain an input method vocabulary that ensures that preset characters with complete semantics will not be separated, the statistical module 730 can be specifically used to: count the frequency of each tuple in the target q-tuple set appearing in the target q+j-tuple set;

[0113] Correspondingly, building block 740 may include:

[0114] A screening unit, used to screen out each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold, to obtain a first candidate tuple; wherein j is a positive integer greater than or equal to 1;

[0115] A construction unit is used to construct an input method vocabulary based on the first candidate tuple.

[0116] In some embodiments of the present application, in order to further obtain an input method vocabulary that ensures that preset characters with complete semantics will not be separated, the construction module 740 may further include:

[0117] A first determining unit is used for removing tuples without textual meaning from the target unit group set to obtain a second candidate tuple;

[0118] Correspondingly, the building blocks can be used for:

[0119] An input method vocabulary is constructed based on the first candidate tuple and the second candidate tuple.

[0120] In some embodiments of the present application, in order to prevent the user from constantly switching input method keyboards when inputting characters in at least two different input modes, the input method word library construction device mentioned above may further include:

[0121] The training module is used to train the language model based on the input method vocabulary to obtain a trained target language model.

[0122] In some embodiments of the present application, the input method word library construction device mentioned above may also include:

[0123] The second processing module is used to perform word segmentation processing on the second corpus based on the target language model to produce a second word segmentation result; wherein the second word segmentation result includes words composed of characters input in at least two input modes.

[0124] The input method word library construction device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0125] The input method word library construction device in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0126] The input method word library construction device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0127] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned vocabulary building method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0128] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0129] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0130] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0131] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0132] Among them, the processor 910 is used to process each word segmentation in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are positive integers greater than 1; and filter out tuple sets including preset characters from each i-tuple set respectively, and obtain a corresponding target i-tuple set; wherein the preset characters include characters input in at least two input modes; and count the frequency of each tuple in the target tuple set; wherein the target tuple set is a set in N i-tuple sets; and determine the target tuple based on the frequency, and build a vocabulary based on the target tuple.

[0133] In this way, by processing each word in the first word segmentation result corresponding to the first corpus, an i-tuple set is obtained, and then a tuple set including preset characters is screened out from each i-tuple set, and corresponding target i-tuple sets are obtained respectively. Since the preset characters include characters input under at least two input modes, the tuples in the target i-tuple set obtained in this way include characters input under at least two input modes. In this way, the input method vocabulary constructed according to the statistical frequency of each tuple in the target tuple set includes characters input under at least two input modes, ensuring that the preset characters with complete semantics will not be separated, and in the process of constructing the vocabulary, a large amount of manpower is not required, thereby improving the efficiency of vocabulary construction.

[0134] Optionally, the processor 910 can be specifically used to: when i=1, form each word in the first word segmentation result into an independent tuple to obtain a single-unit group set; and when i>1, combine each word in the first word segmentation result in the same manner as i adjacent words to obtain a multi-unit group set.

[0135] In this way, when i=1, each word in the first word segmentation result can be formed into an independent tuple to obtain a single-unit group set; when i>1, each word in the first word segmentation result can be combined in the same way as i adjacent words to obtain a multi-unit group set, so that an accurate i-tuple set can be obtained, and then based on the i-tuple set, an input method vocabulary can be reasonably obtained.

[0136] Optionally, the processor 910 can also be specifically used to: count the frequency of each tuple in the target q-tuple set appearing in the target q+j-tuple set; q is a positive integer greater than or equal to 2; filter out each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold, and obtain a first candidate tuple; wherein j is a positive integer greater than or equal to 1; and construct an input method vocabulary based on the first candidate tuple. In this way, the frequency of each tuple in the target q-tuple set appearing in the target q+j-tuple set is counted, and each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold is filtered out to obtain a first candidate tuple; then, based on the first candidate tuple, construct an input method vocabulary. The input method vocabulary constructed in this way can subsequently ensure that preset characters with complete text semantics will not be separated, so as to obtain a more reasonable input method vocabulary, and in the process of constructing the input method vocabulary, no manual participation is required, thereby improving the efficiency of constructing the input method vocabulary.

[0137] Optionally, the processor 910 may also be configured to: for a target unit group set, remove tuples without textual meaning from the target unit group set to obtain a second candidate tuple; and construct an input method vocabulary based on the first candidate tuple and the second candidate tuple.

[0138] In this way, for the target unit group set, tuples without textual meaning are removed from the target unit group set to obtain a second candidate tuple, and then an input method vocabulary is constructed based on the first candidate tuple and the second candidate tuple. The input method vocabulary constructed in this way can subsequently ensure that preset characters with complete textual semantics will not be separated, thereby obtaining a more reasonable input method vocabulary. In addition, in the process of constructing the input method vocabulary, no human participation is required, thereby improving the construction efficiency of the input method vocabulary.

[0139] Optionally, the processor 910 may also be configured to: train a language model based on an input method vocabulary to obtain a trained target language model.

[0140] In this way, the speech model can be trained through the constructed input method vocabulary to obtain a target language model, so that the corpus to be segmented can be subsequently segmented based on the target language model, thereby obtaining a segmentation result in which the preset characters with complete semantics will not be separated.

[0141] Optionally, the processor 910 may also be configured to: perform word segmentation processing on the second corpus based on the target language model to generate a second word segmentation result, wherein the second word segmentation result includes words composed of characters input in at least two input modes.

[0142] In this way, by performing word segmentation processing on the second corpus based on the target language model, the second word segmentation result, since the target language model is constructed based on the input method vocabulary, the second word segmentation result obtained after the word segmentation processing based on the target language model contains words composed of characters input in at least two input modes, thereby achieving the effect that two words with complete semantics will not be separated.

[0143] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0144] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0145] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0146] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned vocabulary library construction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0147] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0148] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned vocabulary library construction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0149] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0150] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned vocabulary building method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0151] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0153] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for constructing a vocabulary library, characterized in that: The method comprises: Process each word in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are positive integers greater than 1; Filtering out a tuple set including preset characters from each i-tuple set respectively, and obtaining a corresponding target i-tuple set; wherein the preset characters include characters input in at least two input modes; Counting the frequency of each tuple in the target tuple set; wherein the target tuple set is a set in N i-tuple sets; Determine a target tuple based on the frequency, and construct a vocabulary based on the target tuple; Count the frequency of each tuple in the target q-tuple set, including: Count the frequency of each tuple in the target q tuple set appearing in the target q+j tuple set; q is a positive integer greater than or equal to 2; The step of determining a target tuple based on the frequency and constructing an input method vocabulary based on the target tuple includes: Filter out each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold, and obtain a first candidate tuple; wherein j is a positive integer greater than or equal to 1; An input method vocabulary is constructed based on the first candidate tuple.

2. The method according to claim 1, characterized in that The step of processing each word in the first word segmentation result to obtain an i-tuple set includes: When i=1, each word in the first word segmentation result is formed into an independent tuple to obtain a unit group set; When i>1, each word segmentation in the first word segmentation result is combined in the same way as i adjacent word segmentations to obtain a multi-tuple set.

3. The method according to claim 1, characterized in that Before constructing the input method word library based on the frequency, the method further includes: For the target unit group set, remove tuples without textual meaning from the target unit group set to obtain a second candidate tuple; The step of constructing an input method vocabulary based on the first candidate tuple includes: An input method vocabulary is constructed based on the first candidate tuple and the second candidate tuple.

4. The method according to any one of claims 1 to 3, characterized in that: After determining the target tuple based on the frequency and constructing the input method vocabulary based on the target tuple, the method further includes: The language model is trained based on the input method vocabulary to obtain a trained target language model.

5. The method according to claim 4, characterized in that After training the language model based on the input method vocabulary to obtain the trained target language model, the method further includes: Performing word segmentation processing on the second corpus based on the target language model to obtain a second word segmentation result; Among them, the second word segmentation result includes words composed of characters input in at least two input modes.

6. A vocabulary building device, characterized in that: The device comprises: The first processing module is used to process each word segmentation in the first word segmentation result corresponding to the first corpus to obtain N i-tuple sets; i∈[1, k], k and N are both positive integers greater than 1; A screening module, used to screen out tuple sets including preset characters from each i-tuple set, and obtain corresponding target i-tuple sets respectively; wherein the preset characters include characters input in at least two input modes; A statistical module, used for counting the frequency of each tuple in a target tuple set; wherein the target tuple set is a set in N i-tuple sets; A construction module, used for determining a target tuple based on the frequency, and constructing a vocabulary based on the target tuple; The statistics module is specifically used for: Count the frequency of each tuple in the target q tuple set appearing in the target q+j tuple set; The building blocks include: A screening unit, used to screen out each tuple in the target q-tuple set whose frequency is greater than or equal to a preset frequency threshold, to obtain a first candidate tuple; wherein j is a positive integer greater than or equal to 1; A construction unit is used to construct an input method vocabulary based on the first candidate tuple.

7. The device according to claim 6, characterized in that The first processing module comprises: A first processing unit is used for, when i=1, forming each word in the first word segmentation result into an independent tuple to obtain a unit group set; The second processing unit is used for combining each word in the first word segmentation result in the manner of combining i adjacent word segmentations to obtain a multi-tuple set when i>1.

8. The device according to claim 6, characterized in that The building block further includes: A first determining unit is used for removing tuples without textual meaning from the target unit group set to obtain a second candidate tuple; The building blocks are specifically used for: An input method vocabulary is constructed based on the first candidate tuple and the second candidate tuple.

9. The device according to any one of claims 6 to 8, characterized in that: The device also includes: The training module is used to train the language model based on the input method vocabulary to obtain a trained target language model.

10. The device according to claim 9, characterized in that The device also includes: The second processing module is used to perform word segmentation processing on the second corpus based on the target language model to produce a second word segmentation result; wherein the second word segmentation result includes words composed of characters input in at least two input modes.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the vocabulary building method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the vocabulary building method according to any one of claims 1 to 5 are implemented.

Images