A word segmentation method and server

Abstract

The embodiment of the present application discloses a word segmentation method and server, which can segment a character string composed of an English string, a pinyin string or a symbol string into multiple segments, obtain an effective word segmentation result, and facilitate the normal progress of subsequent steps of natural language processing. The method of the embodiment of the present application includes: obtaining a character string to be segmented, the character string to be segmented belongs to the American Standard Code for Information Interchange ASCII character string; performing word segmentation processing on the character string to be segmented by regular expression matching to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, the first ASCII character string includes M separated segments, and M is an integer greater than 1; performing word segmentation processing on the first ASCII character string by a hidden Markov HMM model to obtain a second ASCII character string, wherein the HMM model is obtained by training with an ASCII character string, and the second ASCII character string includes N separated segments, and N is an integer greater than M.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a word segmentation method and a server. Background Art

[0002] With the development of Internet technology, word segmentation technology, as an important basic technology in the field of natural language processing, is the first step to realize machine recognition of human language. Therefore, word segmentation technology is widely used in application branches of natural language processing such as text-to-speech conversion, machine translation, speech recognition, text summarization, and text retrieval.

[0003] Word segmentation is the process of dividing a sentence into individual words, and recombining continuous sentences into word sequences according to certain specifications. Taking Chinese word segmentation technology as an example, the goal of Chinese word segmentation technology is to divide a sentence into individual Chinese words, so that the machine can recognize human language.

[0004] In the prior art, although there are many word segmentation schemes, most of these word segmentation schemes are for Chinese. However, in some scenarios, we often encounter strings composed of English strings, Pinyin strings or symbol strings, and there is currently no good word segmentation scheme for these strings. The existing processing method usually treats these strings as one word without performing more detailed segmentation, which easily affects the subsequent steps of natural language processing. Summary of the invention

[0005] The embodiment of the present application provides a word segmentation method, which can divide a character string composed of an English string, a pinyin string or a symbol string into multiple segments, obtain an effective word segmentation result, and facilitate the normal progress of subsequent steps of natural language processing.

[0006] In view of this, a first aspect of an embodiment of the present application provides a word segmentation method, comprising:

[0007] Get the string to be segmented, which is an ASCII string of the American Standard Code for Information Interchange;

[0008] Perform word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M delimited segments, where M is an integer greater than 1;

[0009] The first ASCII character string is segmented by a hidden Markov model to obtain a second ASCII character string, wherein the HMM model is obtained by training with the ASCII character string, and the second ASCII character string includes N separated segments, where N is an integer greater than M.

[0010] A second aspect of the present application provides a server, including:

[0011] An acquisition unit is used to acquire a character string to be segmented, where the character string to be segmented is an American Standard Code for Information Interchange ASCII character string;

[0012] A processing unit, configured to perform word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M delimited segments, where M is an integer greater than 1;

[0013] The processing unit is further used to perform word segmentation processing on the first ASCII string through a hidden Markov model HMM to obtain a second ASCII string, wherein the HMM model is obtained by training with the ASCII string, and the second ASCII string includes N separated segments, where N is an integer greater than M.

[0014] In a possible design, in a first implementation manner of the second aspect of the embodiments of the present application, a training unit is further included;

[0015] The acquiring unit is further used to acquire the first text information;

[0016] The processing unit is further used to perform word segmentation processing on the first text information to obtain text information to be processed;

[0017] The processing unit is further used to determine target text information according to the text information to be processed, wherein the target text information includes at least one voice element, and each voice element has a corresponding relationship with a character in the text information to be processed, and the voice element is an ASCII character or an ASCII character string;

[0018] A training unit, used for training the HMM model through target text information to obtain a trained HMM model;

[0019] The processing unit is further used to perform word segmentation processing on the first ASCII character string through the trained HMM model to obtain a second ASCII character string.

[0020] In a possible design, in a second implementation of the second aspect of the embodiment of the present application,

[0021] The acquiring unit is further used to acquire second text information from the first text database, wherein the second text information has a separator, the separator is used to perform word segmentation processing on the second text information, and the second text information is composed of ASCII characters;

[0022] The training unit is further used to train the HMM model through the second text information to obtain a trained HMM model;

[0023] The processing unit is further used to perform word segmentation processing on the first ASCII character string through the trained HMM model to obtain a second ASCII character string.

[0024] In a possible design, in a third implementation manner of the second aspect of the embodiment of the present application, an adjustment unit is further included;

[0025] The acquisition unit is also used to acquire the preset adjustment rules;

[0026] The adjustment unit is used to adjust the word segmentation result of the second ASCII character string according to a preset adjustment rule to obtain a third ASCII character string, wherein the preset adjustment rule is used to combine or re-separate the separated segments in the second ASCII character string.

[0027] In a possible design, in a fourth implementation of the second aspect of the embodiments of the present application,

[0028] The adjusting unit is specifically configured to combine the multiple continuous first separation segments to obtain a third ASCII character string if there are multiple continuous first separation segments in the second ASCII character string, wherein the first separation segments only contain a single character.

[0029] In a possible design, in a fifth implementation of the second aspect of the embodiments of the present application,

[0030] an adjustment unit, specifically configured to combine the second separation segment with the previous separation segment to obtain a third ASCII string if the second separation segment exists in the second ASCII string;

[0031] The second separated segment includes characters or character strings in the first preset vocabulary.

[0032] In a possible design, in a sixth implementation of the second aspect of the embodiments of the present application,

[0033] an adjusting unit, specifically configured to separate the third separation segment into at least two fourth separation segments if the third separation segment exists in the second ASCII character string, so as to obtain a third ASCII character string;

[0034] The third separated segment includes a character string in the second preset vocabulary.

[0035] In a possible design, in a seventh implementation of the second aspect of the embodiments of the present application,

[0036] The adjustment unit is further used to combine the separated segments in the third ASCII character string according to the second text database to obtain a fourth ASCII character string, and the second text database includes at least one custom-obtained character string.

[0037] A third aspect of the present application provides a server, including: a memory, a transceiver, a processor, and a bus system;

[0038] Wherein, the memory is used to store programs;

[0039] The processor is used to execute the program in the memory, including the following steps:

[0040] Get the string to be segmented, which is an ASCII string of the American Standard Code for Information Interchange;

[0041] Perform word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M delimited segments, where M is an integer greater than 1;

[0042] Using a hidden Markov model (HMM), the first ASCII character string is segmented to obtain a second ASCII character string, wherein the HMM model is obtained by training the ASCII character string, and the second ASCII character string includes N separated segments, where N is an integer greater than M;

[0043] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0044] A fourth aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes any of the methods described above.

[0045] A fifth aspect of the present application provides a computer program product comprising instructions, which, when executed on a computer or a processor, enables the computer or the processor to execute any of the methods described above.

[0046] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0047] In the embodiment of the present application, for an ASCII character string composed of an English string, a Pinyin string or a symbol string, the preliminary segmentation boundaries of the ASCII character string are first determined by regular expression matching, and then the ASCII character string is further segmented according to the HMM model to obtain an effective segmentation result. Since the training data used by the HMM model is composed of character strings of the same type as the ASCII character string, the ASCII character string can be segmented in a targeted manner, thereby ensuring the accuracy of the segmentation processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 An exemplary system architecture of an embodiment of the method for word segmentation that can be applied to the present application is shown;

[0049] Figure 2 The flowchart of an embodiment of a word segmentation method of an embodiment of the present application is shown;

[0050] Figure 3 A flowchart of a word segmentation method in an application scenario of an embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of an embodiment of a server in an embodiment of the present application;

[0052] Figure 5 This is another schematic diagram of a server in an embodiment of the present application;

[0053] Figure 6 This is another schematic diagram of a server in an embodiment of the present application;

[0054] Figure 7 It is a schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiment of the present application provides a word segmentation method and server, which can divide a character string composed of an English string, a pinyin string or a symbol string into multiple segments, obtain an effective word segmentation result, and facilitate the normal implementation of subsequent steps of natural language processing.

[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] Figure 1 An exemplary system architecture of an embodiment of the method for word segmentation that can be applied to the present application is shown.

[0058] like Figure 1 As shown, the system architecture may include a terminal device, a network and a server. The network is used to provide a medium for a communication link between the terminal device and the server. The network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0059] Users can use terminal devices to interact with servers through the network to receive or send messages, etc. Various communication client applications can be installed on the terminal devices, such as language processing software, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0060] The terminal device can be hardware or software. When the terminal device is hardware, it can be various electronic devices with display screens, including but not limited to smart phones, PDAs, tablet computers, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, laptop computers and desktop computers, etc. When the terminal device is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0061] The server may be a server providing various services, such as a text processing server that performs word segmentation on a character string to be segmented sent by a terminal device. The text processing server may perform analysis and other processing on the received data such as the character string to be segmented to obtain a processing result (such as a word segmentation result).

[0062] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers may be provided as required. In the case where the data used in the process of generating the segmentation result corresponding to the text to be segmented or the text to be segmented does not need to be obtained remotely, the above system architecture may not include a network, but only include terminal devices or servers.

[0064] The following will introduce the word segmentation method in the embodiment of the present application from the perspective of the server. Figure 2 , Figure 2 The flow chart of an embodiment of a word segmentation method in an embodiment of the present application is shown; an embodiment of the word segmentation method in an embodiment of the present application includes:

[0065] 201. Obtain a character string to be segmented, where the character string to be segmented is an American Standard Code for Information Interchange ASCII character string;

[0066] In this embodiment, the execution subject of the method for word segmentation (for example Figure 1 The server shown in FIG. 1 can be connected to the terminal (eg, Figure 1 The terminal device shown in the figure) or locally obtains the character string to be segmented. For example, when a user needs to search the meaning of a character string through a search engine, the character string can be input on the terminal, and the terminal sends the character string to the server; after the server obtains the character string from the terminal, in order to be able to recognize the character string, the character string can be regarded as the character string to be segmented, and then the character string to be segmented is subjected to subsequent segmentation processing to obtain the segmentation result.

[0067] Among them, the character string to be segmented belongs to the American Standard Code for Information Interchange (ASCII) character string. It is understandable that in a computer, all data must be represented by binary numbers when stored and calculated. For example, 52 letters such as a, b, c, d, A, B, C, D, and numbers such as 0 and 1, as well as some commonly used symbols (such as *, #, @, etc.) must also be represented by binary numbers when stored in a computer. There can be many different standards for which binary numbers are used to represent which symbols, and ASCII is a set of standards that uniformly stipulate which binary numbers are used to represent the above commonly used symbols. ASCII uses 7-bit binary numbers to represent all uppercase and lowercase letters, numbers 0 to 9, punctuation marks, and some special control characters. The ASCII character string is composed of characters that can be represented by ASCII, including English character strings, number character strings, symbol character strings, and pinyin character strings. For example, "iloveyou", "5201314", "woaibeijingtiananmen", "zhangsan(zhangsan@qq.com)lovesbeijingtiananmen", etc. are all ASCII character strings.

[0068] 202. Perform word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M delimited segments, where M is an integer greater than 1;

[0069] Since the character string to be segmented may usually include English characters, numeric characters, and symbol characters mixed together, and some specific English characters, numeric characters, or symbol characters have specific meanings when combined together, it is usually hoped that these specific combinations can be identified to avoid combining these specific combinations with other parts of the character string to be segmented, thereby affecting the segmentation result. Therefore, in this embodiment, the character string to be segmented can be first segmented by regular expression matching, and some specific characters or character combinations in the character string to be segmented can be identified and separated from other parts of the character string to be segmented.

[0070] Among them, a regular expression is a "regular string" composed of some pre-defined specific characters and combinations of these specific characters. For example, a regular expression can specifically include a Uniform Resource Locator (URL), such as "www.qq.com", "www.google.com", etc.; an electronic mail (Electronic Mail, E-mail) address, such as "zhangsan@qq.com", "lisi@163.com", etc.; an Internet Protocol (Internet Protocol, IP) address, such as "10.0.0.1", "198.168.1.1", etc.; a floating point (Float) number, such as "3.14", "-5.23", etc.; a separator, such as "?", "|", ".", "(" ")", etc. The above is just an example of the character strings that a regular expression can include, and it does not mean that a regular expression only includes the above-listed character strings. A regular expression can also include other specific character strings, which will not be repeated here. Take the string "zhangsan(zhangsan@qq.com)lovesbeijingtiananmen" as an example. After segmenting it with regular expression matching, the segmentation result is "zhangsan / ( / zhangsan@qq.com / ) / lovesbeijingtiananmen". Obviously, the brackets in the string and the complete E-mail address in the brackets can be recognized and separated from other parts. The segmentation result contains the following 5 separated segments: "zhangsan", "(", zhangsan@qq.com, ")", "lovesbeijingtiananmen".

[0071] In this embodiment, the character string to be segmented is firstly segmented by regular expression matching, which can identify some specific character string patterns in the character string to be segmented, and separate these specific character string patterns from other parts, which has the function of determining the preliminary division boundary and can ensure the accuracy of subsequent segmentation results.

[0072] 203. Perform word segmentation processing on the first ASCII character string through a hidden Markov model (HMM) to obtain a second ASCII character string, wherein the HMM model is obtained by training with the ASCII character string, and the second ASCII character string includes N separated segments, where N is an integer greater than M.

[0073] In this embodiment, the word segmentation result obtained after the word segmentation processing by regular expression matching can usually only identify some specific string patterns. There may be some English strings, pinyin strings, etc. that need to be further subdivided in the word segmentation result, such as "iloveyou", "woaini", etc. For these strings, the semantically related Hidden Markov Model (HMM) model can be used for segmentation.

[0074] The HMM model requires a five-tuple, namely: state value set, observation value set, initial state distribution, transition probability matrix, and emission probability matrix.

[0075] The state value set is (B, M, E, S), {B: begin, M: middle, E: end, S: single}, which refers to different state values ​​of the character respectively. The state value represents the position of the character in the word. The state value B represents that the character is the starting word in the word, the state value M represents that the character is the middle word in the word, the state value E represents that the character is the ending word in the word, and the state value S represents that the character is a single word.

[0076] The observation value set is the set of all ASCII characters (a, b, c, A, B, C...), even including all half-width punctuation marks.

[0077] The state value is the value obtained by performing operations through the HMM model. In the word segmentation processing of the ASCII string through the HMM model, the input of the HMM model is the first ASCII string (that is, the observation value set), and the output of the HMM model is the state value of each character in the first ASCII string. Based on the state value of each character in the first ASCII string, the first ASCII string can be segmented to obtain further word segmentation results (that is, the second ASCII string).

[0078] In this embodiment, the state value can be predicted by using the Viterbi algorithm, wherein the observation value is the input of the Viterbi algorithm, and the sequence value is the output of the Viterbi algorithm. Between the input and the output, the Viterbi algorithm also needs to use three model parameters, namely, the initial state distribution, the transition probability matrix, and the emission probability matrix.

[0079] The initial state distribution is the probability that the first character in the string belongs to the four states {B, E, M, S}. According to experience, the probability that the first character in the string belongs to E and M is 0. Therefore, the first character at the beginning of the string can only be the beginning of a word (B) or a single word (S), that is, it belongs to one of the two states B or S.

[0080] The transition probability matrix is ​​a 4×4 (4 refers to the size of the state set) matrix. Each value in it represents the probability of transitioning from one state to another. There will also be many values ​​​​of 0 in the transition probability matrix, such as the probability value of transitioning from B to S, the probability value of transitioning from S to M, etc.

[0081] The emission probability matrix refers to the probability distribution from each state value to each observation value. According to the independence assumption of observation values, the observation value only depends on the current state value.

[0082] The acquisition of the initial state distribution, transition probability matrix, and emission probability matrix is ​​the training part of the HMM model. Since the ASCII string includes both English strings and Pinyin strings, as well as half-width punctuation marks, when selecting training data, the segmentation results of English strings, Pinyin strings, and half-width punctuation marks can be selected as training data to ensure the word segmentation effect of the HMM model. By selecting the above training data, the real segmentation results of the ASCII string can be obtained, that is, the observation value sequence. The state values ​​of the observation values ​​in these sequences are marked, and statistics are performed to obtain three probability matrices.

[0083] After the HMM model is trained, the trained HMM model can be used to perform word segmentation on the ASCII string. The specific steps are as follows:

[0084] 1) Traverse the ASCII string from beginning to end and calculate the probability of the last character in the ASCII string corresponding to each of the four state values. The state value of the last character in the ASCII string can only be E or S, and cannot be M or B. Therefore, only the probabilities corresponding to the state value of the last character being E or S are compared to obtain the maximum value, thereby determining the state value of the last character.

[0085] 2) Get the status value of each character in the ASCII string by backtracking;

[0086] 3) The ASCII string is segmented according to the status value of each character in the ASCII string. Since both the ending E and the ending S mark the end of a word, the parts ending with E and S in the ASCII string are segmented to obtain the specific segmentation method of the ASCII string.

[0087] In this embodiment, for an ASCII character string composed of an English string, a Pinyin string or a symbol string, the preliminary segmentation boundary of the ASCII character string is first determined by regular expression matching, and then the ASCII character string is further segmented according to the HMM model to obtain an effective segmentation result. Since the training data used by the HMM model is composed of character strings of the same type as the ASCII character string, the ASCII character string can be segmented in a targeted manner, thereby ensuring the accuracy of the segmentation processing.

[0088] Optionally, in this embodiment, after the character string to be segmented is segmented by regular expression matching, the character substring that hits the regular expression match can be further segmented and matched, so that the segmentation result is more detailed, and some information in the character string can be further mined. Specifically, the process of segmenting the character substring can be to use commonly used separators (such as ".", "·", etc.) to further segment the character substring. Take the character string "www.qq.comlisilovesbeijing" as an example. After the regular expression matching is performed, the segmentation result of the string is "www.qq.com / lisilovesbeijing"; wherein, the character substring "www.qq.com" hits the regular expression match, and after further segmentation matching, it can be segmented into: "www / . / qq / . / com". In actual applications, users can choose whether to perform further segmentation matching in regular expression matching according to actual needs. Specifically, after the user enters a character string on the search engine of the terminal, he can choose to turn on the segmentation matching switch of the search engine, so that the server corresponding to the search engine continues to perform segmentation matching on the character substrings in the string that hit the regular expression match after performing regular expression matching on the character string.

[0089] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in a first optional embodiment of the word segmentation method provided in the embodiments of the present application, before performing word segmentation processing on the first ASCII string through the HMM model to obtain the second ASCII string, the method further includes:

[0090] Obtaining first text information;

[0091] Performing word segmentation processing on the first text information to obtain text information to be processed;

[0092] Determining target text information according to the text information to be processed, wherein the target text information includes at least one voice element, and each voice element has a corresponding relationship with a character in the text information to be processed, and the voice element is an ASCII character or an ASCII character string;

[0093] The HMM model is trained through the target text information to obtain a trained HMM model;

[0094] The first ASCII string is segmented by the HMM model to obtain a second ASCII string, including:

[0095] The trained HMM model is used to perform word segmentation on the first ASCII character string to obtain a second ASCII character string.

[0096] In this embodiment, the first text information may be text information in Chinese, Japanese, etc. The first text information is composed of non-ASCII characters, but the first text information can be converted into target text information composed of ASCII characters. Taking the first text information as Chinese text information as an example, after obtaining the Chinese text information, it can be segmented based on some existing Chinese word segmentation schemes to obtain the Chinese text information to be processed; then, the Chinese text information to be processed is converted into pinyin text information, thereby obtaining the target text information. Specifically, taking the first text information as "I love Beijing" as an example, it is segmented to obtain the text information to be processed as "I / love / Beijing"; then, the characters in the text information to be processed are converted into phonetic elements belonging to ASCII characters or ASCII character strings, that is, the Chinese characters in the text information to be processed are converted into pinyin characters, so that the target text information after segmentation is "wo / ai / beijing". Since the target text information is composed of ASCII characters, the target text information can be used as training data to train the HMM model.

[0097] Secondly, in this embodiment, after obtaining the first text information, the first text information is first segmented according to the existing segmentation scheme. After obtaining the text information to be processed, the text information to be processed is converted into the corresponding target text information composed of ASCII character strings. Finally, the training data of the HMM model trained with the target text information is used to train the HMM model. This can ensure that the HMM model can segment the character strings converted from text information in languages ​​other than English, thereby improving the applicability of the HMM model for word segmentation processing.

[0098] Optionally, in the above Figure 2 On the basis of the corresponding embodiment, in a second optional embodiment of the word segmentation method provided in the embodiment of the present application, before performing word segmentation processing on the first ASCII string through the HMM model to obtain the second ASCII string, the method further includes:

[0099] Acquire second text information from the first text database, wherein the second text information has a separator, the separator is used to perform word segmentation processing on the second text information, and the second text information is composed of ASCII characters;

[0100] The HMM model is trained by using the second text information to obtain a trained HMM model;

[0101] The first ASCII string is segmented by the HMM model to obtain a second ASCII string, including:

[0102] The trained HMM model is used to perform word segmentation on the first ASCII character string to obtain a second ASCII character string.

[0103] In this embodiment, the first text database may specifically include a database containing a large amount of English text information, such as a Wiki system and an English news database. The second text information (i.e., English text information) composed of ASCII characters can be directly obtained from the first text database. Since the second text information usually contains natural separators for word segmentation (such as spaces, commas, periods, etc.), the second text information can actually be regarded as text information that has been word segmented and can be directly used as training data to train the HMM model. Taking the second text information "I am from England, but I love China" as an example, since the second text information contains separators such as spaces and commas, the second text information can actually be regarded as the following text information that has been word segmented "I / am / from / England / but / i / love / China", so the second text information can be directly used to train the HMM model.

[0104] Secondly, in this embodiment, by directly obtaining the second text information with separators from the first text database, the second text information is used as text information that has been segmented to train the HMM model, and the trained HMM model is obtained, thereby ensuring that the HMM can segment the ASCII character string based on the semantic correlation between words and obtain accurate segmentation results.

[0105] Optionally, in the above Figure 2 On the basis of the corresponding embodiment, in a third optional embodiment of the word segmentation method provided in the embodiment of the present application, after performing word segmentation processing on the first ASCII string through the HMM model to obtain the second ASCII string, the method further includes:

[0106] Get preset adjustment rules;

[0107] The word segmentation result of the second ASCII character string is adjusted according to a preset adjustment rule to obtain a third ASCII character string, wherein the preset adjustment rule is used to combine or re-separate the separated segments in the second ASCII character string.

[0108] It is understandable that the word segmentation result obtained by performing word segmentation processing through the HMM model is mainly based on the training status of the HMM model. Since the training data used to train the HMM model is limited, the word segmentation processing using the HMM model will also have certain limitations. In some cases, the word segmentation result obtained by performing word segmentation processing through the HMM model may not be a word segmentation result that meets the requirements. In these cases, the word segmentation result needs to be adjusted again so that the word segmentation result finally obtained is a word segmentation result that meets the requirements. Specifically, after performing word segmentation processing on the first ASCII character string through the HMM model to obtain the second ASCII character string, the preset adjustment rule established by the user is obtained, and then the word segmentation result of the second ASCII character string is adjusted according to the preset adjustment rule to obtain a third ASCII character string that meets the requirements. For example, taking the second ASCII character string "shenzhen / huanle / gu" as an example, after the word segmentation processing of the HMM model, the character substring "huanlegu" in the second ASCII character string is segmented into "huanle / gu", and this segmentation result does not meet the requirements. In fact, "huanlegu" is the segmentation result that meets the requirements. Therefore, at this time, the second ASCII character string "shenzhen / huanle / gu" can be adjusted to the third ASCII character string "shenzhen / huanlegu" through the preset adjustment rule.

[0109] Secondly, in this embodiment, by obtaining preset adjustment rules and adjusting the word segmentation results obtained by word segmentation processing through the HMM model according to the preset adjustment rules, the word segmentation results obtained based on the HMM model can be corrected to ensure the correctness and personalization of the final word segmentation results.

[0110] Optionally, based on the third optional embodiment described above, in a fourth optional embodiment of the word segmentation method provided in an embodiment of the present application, the word segmentation result of the second ASCII character string is adjusted according to a preset adjustment rule to obtain a third ASCII character string, including:

[0111] If there are multiple continuous first separation segments in the second ASCII character string, the multiple continuous first separation segments are combined to obtain a third ASCII character string, wherein the first separation segments only contain a single character.

[0112] It is understandable that, generally speaking, only a few English letters such as "i" and "u" have actual meanings, while most English letters have no particular actual meanings. Therefore, when there are multiple consecutive single letters in the second ASCII string, in order to avoid too many separators due to too fine word segmentation granularity, these consecutive single letters can be combined, which is equivalent to combining these consecutive single letters that have no actual meaning together. For example, when the word segmentation result of the second ASCII string contains a string such as "b / c / d / e" composed of multiple consecutive first separator segments "b", "c", "d", and "e", the above multiple consecutive first separator segments can be combined together to form a single separator segment "bcde", thereby obtaining a third ASCII string with more reasonable word segmentation.

[0113] Secondly, in this embodiment, when there are multiple continuous first separation segments without actual meaning in the second ASCII character string, the multiple continuous first separation segments are combined into a single separation segment, thereby ensuring that the word segmentation granularity is not too fine and ensuring the accuracy of the word segmentation result.

[0114] Optionally, based on the third optional embodiment described above, in a fifth optional embodiment of the word segmentation method provided in an embodiment of the present application, the word segmentation result of the second ASCII character string is adjusted according to a preset adjustment rule to obtain a third ASCII character string, including:

[0115] If the second delimited segment exists in the second ASCII character string, the second delimited segment is combined with the previous delimited segment to obtain a third ASCII character string;

[0116] The second separated segment includes characters or character strings in the first preset vocabulary.

[0117] It is understandable that in English text, the character "s" is generally used as the end of a noun or verb, indicating the plural or third person form, such as "cars", "loves", etc.; the characters "d", "ed", "ing", etc. are generally used as the end of a verb, indicating the past tense or the present tense of the verb, such as "danced", "looked", "doing", etc.; therefore, these characters actually need to be combined with the words preceding them, and do not need to be segmented separately. However, in some cases, due to the relationship between training data, the above characters are usually segmented separately. Therefore, in these cases, the word segmentation results need to be adjusted so that these characters can be combined with the words preceding them.

[0118] In this embodiment, a first preset word library may be pre-established, and the first preset word library may specifically include characters or strings such as "s", "d", "ed", and "ing". When a second separator segment matching a character or string in the first preset word library exists in the second ASCII string, the second separator segment is combined with a previous segment of the second separator segment as a new separator segment, thereby obtaining a third ASCII string. For example, when the second ASCII string is "li / si / love / s / beijing", the second separator segment "s" is combined with the previous segment "love" to obtain a new separator segment "loves", thereby obtaining a third ASCII string of "li / si / loves / beijing".

[0119] Secondly, in this embodiment, by setting the first preset vocabulary, when there is a second separator segment including characters or strings in the first preset vocabulary in the second ASCII character string, the second separator segment is combined with the previous separator segment to obtain a third ASCII character string, which can adjust the word segmentation result and improve the accuracy of word segmentation.

[0120] Optionally, based on the third optional embodiment described above, in a sixth optional embodiment of the word segmentation method provided in an embodiment of the present application, the word segmentation result of the second ASCII character string is adjusted according to a preset adjustment rule to obtain a third ASCII character string, including:

[0121] If the third separation segment exists in the second ASCII character string, the third separation segment is separated into at least two fourth separation segments to obtain a third ASCII character string;

[0122] The third separated segment includes a character string in the second preset vocabulary.

[0123] It is understandable that in some specific fields, there are usually some common word combinations. For these word combinations, the HMM model may divide them into a separated segment due to inability to recognize them, but in fact these word combinations can be further segmented to facilitate the subsequent natural language processing steps. For example, for the string "tiananmen", the word segmentation processing performed by the HMM model usually treats the string as a separate separated segment, but in fact the string can also be divided into "tianan / men"; for the string "dongfangmingzhu", the word segmentation processing performed by the HMM model usually treats the string as a separate separated segment, but in fact the string can also be divided into "dongfang / mingzhu". Therefore, in order to further segment some specific combination words, a second preset vocabulary can be established in advance, which specifically includes some specific character strings, such as "tiananmen", "dongfangmingzhu", etc. In addition, each character string in the second preset vocabulary has a corresponding separation rule. When there is a third separation segment in the second ASCII character string corresponding to the character string in the second preset vocabulary, the third separation segment is separated into at least two fourth separation segments based on the separation rule corresponding to the character string, thereby obtaining a third ASCII character string.

[0124] Secondly, in this embodiment, by setting a second preset vocabulary, when the second ASCII character string contains a third separated segment corresponding to a character string in the second preset vocabulary, the third separated segment is separated into at least two fourth separated segments according to the separation rule of the character string corresponding to it, thereby obtaining a third ASCII character string, which can adjust the word segmentation result and improve the accuracy of word segmentation.

[0125] Optionally, based on the third optional embodiment described above, in a seventh optional embodiment of the word segmentation method provided in an embodiment of the present application, after adjusting the word segmentation result of the second ASCII character string according to a preset adjustment rule to obtain a third ASCII character string, the method further includes:

[0126] The separated segments in the third ASCII character string are combined according to the second text database to obtain a fourth ASCII character string, wherein the second text database includes at least one custom-obtained character string.

[0127] In this embodiment, the second text database can be a custom dictionary, and the second text database includes at least one character string pre-customized by the user, such as "beijing", "shanghai", "guangzhou", etc. In some specific fields, there are usually some common compound words. For these compound words, the HMM model may separate them into multiple separated segments, for example, "beijing" is divided into "bei / jing", "shanghai" is divided into "shang / hai" or "guangzhou" is divided into "guang / zhou", etc. However, for these compound words, after they are separated into multiple separated segments, it is usually difficult to express their actual meaning. Therefore, for these words separated into multiple separated segments by the HMM model, the multiple separated segments can be recombined together. Specifically, the multiple separated segments in the third ASCII character string that can be combined into the character string in the second text database can be combined by matching the second text database, so as to obtain a fourth ASCII character string after the word segmentation result is adjusted.

[0128] Secondly, in this embodiment, by setting up a second text database, when there are multiple separated segments in the third ASCII character string that can be combined into a character string in the second text database, these separated segments are recombined into one separated segment, so as to obtain a fourth ASCII character string after adjusting the word segmentation result, which can further adjust the word segmentation result and improve the accuracy of word segmentation.

[0129] For ease of understanding, the following will combine Figure 3 Introducing an application scenario of word segmentation method, please refer to Figure 3 , Figure 3 The following is a flow chart of a word segmentation method in an application scenario of an embodiment of the present application, as shown in the figure, specifically:

[0130] S1. First, obtain the character string to be segmented input by the user;

[0131] S2, performing word segmentation processing on the character string to be segmented by matching regular expressions, identifying some specific character strings in the character string to be segmented, preliminarily dividing the segmentation boundaries of the character string to be segmented, and obtaining a first ASCII character string;

[0132] S3, determine whether the user has turned on the switch for segment matching, if so, go to step S4, otherwise, go to step S5;

[0133] S4, performing segmentation matching on the first ASCII character string, by identifying separators (such as ".", "·", etc.) in the first ASCII character string, and further segmenting the first ASCII character string according to the separators;

[0134] S5, performing word segmentation processing on the first ASCII string by using the HMM model, segmenting the first ASCII string based on the semantics of characters in the first ASCII string, and obtaining a second ASCII string;

[0135] S6, determining whether the user has turned on the switch for adjusting the word segmentation results according to the preset adjustment rules, if so, go to step S7, otherwise, go to step S10;

[0136] S7, adjusting the word segmentation result of the second ASCII character string according to a preset adjustment rule, specifically, including: combining multiple consecutive single characters in the word segmentation result; combining some characters or character strings in the word segmentation result with the previous separation fragments; performing secondary separation on some separation fragments in the word segmentation result;

[0137] S8, determining whether the user has turned on the switch for adjusting the word segmentation results according to the custom dictionary, if so, go to step S9, otherwise, go to step S10;

[0138] S9, adjusting the word segmentation result according to the custom dictionary, combining some separated fragments in the word segmentation result to obtain a new word segmentation result;

[0139] S10. Output the word segmentation result obtained after word segmentation processing.

[0140] Take the string to be segmented as "zhangsan(zhangsan@qq.com)lovesbeijingtiananmen" as an example. Figure 3 The word segmentation results obtained by the word segmentation process shown are exemplified. Please refer to Table 1, which is a comparison table of word segmentation results obtained based on different word segmentation steps.

[0141] Table 1 Comparison of word segmentation results

[0142]

[0143] The server 400 in this application is described in detail below. Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the server 400 in the embodiment of the present application, including:

[0144] An acquisition unit 401 is used to acquire a character string to be segmented, where the character string to be segmented is an American Standard Code for Information Interchange ASCII character string;

[0145] A processing unit 402 is configured to perform word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M delimited segments, where M is an integer greater than 1;

[0146] The processing unit 402 is further configured to perform word segmentation processing on the first ASCII string through a hidden Markov model (HMM) to obtain a second ASCII string, wherein the HMM model is obtained by training with the ASCII string, and the second ASCII string includes N separated segments, where N is an integer greater than M.

[0147] In this embodiment, for an ASCII character string composed of an English string, a Pinyin string or a symbol string, the preliminary segmentation boundary of the ASCII character string is first determined by regular expression matching, and then the ASCII character string is further segmented according to the HMM model to obtain an effective segmentation result. Since the training data used by the HMM model is composed of character strings of the same type as the ASCII character string, the ASCII character string can be segmented in a targeted manner, thereby ensuring the accuracy of the segmentation processing.

[0148] Optionally, in the above Figure 4 Based on the corresponding embodiments, please refer to Figure 5 , in another embodiment of the server 400 provided in the embodiment of the present application, further comprising: a training unit 403;

[0149] The acquisition unit 401 is further used to acquire the first text information;

[0150] The processing unit 402 is further configured to perform word segmentation processing on the first text information to obtain text information to be processed;

[0151] The processing unit 402 is further configured to determine target text information according to the text information to be processed, wherein the target text information includes at least one voice element, and each voice element has a corresponding relationship with a character in the text information to be processed, and the voice element is an ASCII character or an ASCII character string;

[0152] A training unit 403 is used to train the HMM model using target text information to obtain a trained HMM model;

[0153] The processing unit 402 is further configured to perform word segmentation processing on the first ASCII character string by using the trained HMM model to obtain a second ASCII character string.

[0154] Secondly, in this embodiment, after obtaining the first text information, the first text information is first segmented according to the existing segmentation scheme. After obtaining the text information to be processed, the text information to be processed is converted into the corresponding target text information composed of ASCII character strings. Finally, the training data of the HMM model trained with the target text information is used to train the HMM model. This can ensure that the HMM model can segment the character strings converted from text information in languages ​​other than English, thereby improving the applicability of the HMM model for word segmentation processing.

[0155] Optionally, in the above Figure 4 On the basis of the corresponding embodiment, in another embodiment of the server 400 provided in the embodiment of the present application,

[0156] The acquisition unit 401 is further used to acquire second text information from the first text database, wherein the second text information has a separator, the separator is used to perform word segmentation processing on the second text information, and the second text information is composed of ASCII characters;

[0157] The training unit 403 is further used to train the HMM model through the second text information to obtain a trained HMM model;

[0158] The processing unit 402 is further configured to perform word segmentation processing on the first ASCII character string by using the trained HMM model to obtain a second ASCII character string.

[0159] Secondly, in this embodiment, by directly obtaining the second text information with separators from the first text database, the second text information is used as text information that has been segmented to train the HMM model, and the trained HMM model is obtained, thereby ensuring that the HMM can segment the ASCII character string based on the semantic correlation between words and obtain accurate segmentation results.

[0160] Optionally, in the above Figure 4 Based on the corresponding embodiments, please refer to Figure 6 In another embodiment of the server 400 provided in the embodiment of the present application, an adjustment unit 404 is further included;

[0161] The acquisition unit 401 is also used to acquire the preset adjustment rules;

[0162] The adjusting unit 404 is configured to adjust the word segmentation result of the second ASCII character string according to a preset adjustment rule to obtain a third ASCII character string, wherein the preset adjustment rule is used to combine or re-separate the separated segments in the second ASCII character string.

[0163] Secondly, in this embodiment, by obtaining preset adjustment rules and adjusting the word segmentation results obtained by word segmentation processing through the HMM model according to the preset adjustment rules, the word segmentation results obtained based on the HMM model can be corrected to ensure the correctness and personalization of the final word segmentation results.

[0164] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the server 400 provided in the embodiment of the present application,

[0165] The adjusting unit 404 is specifically configured to combine the multiple consecutive first separation segments to obtain a third ASCII character string if there are multiple consecutive first separation segments in the second ASCII character string, wherein the first separation segments only contain a single character.

[0166] Secondly, in this embodiment, when there are multiple continuous first separation segments without actual meaning in the second ASCII character string, the multiple continuous first separation segments are combined into a single separation segment, thereby ensuring that the word segmentation granularity is not too fine and ensuring the accuracy of the word segmentation result.

[0167] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the server 400 provided in the embodiment of the present application,

[0168] The adjusting unit 404 is specifically configured to combine the second separation segment with the previous separation segment to obtain a third ASCII character string if the second separation segment exists in the second ASCII character string;

[0169] The second separated segment includes characters or character strings in the first preset vocabulary.

[0170] Secondly, in this embodiment, by setting the first preset vocabulary, when there is a second separator segment including characters or strings in the first preset vocabulary in the second ASCII character string, the second separator segment is combined with the previous separator segment to obtain a third ASCII character string, which can adjust the word segmentation result and improve the accuracy of word segmentation.

[0171] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the server 400 provided in the embodiment of the present application,

[0172] The adjusting unit 404 is specifically configured to separate the third separation segment into at least two fourth separation segments if the third separation segment exists in the second ASCII character string, so as to obtain a third ASCII character string;

[0173] The third separated segment includes a character string in the second preset vocabulary.

[0174] Secondly, in this embodiment, by setting a second preset vocabulary, when the second ASCII character string contains a third separated segment corresponding to a character string in the second preset vocabulary, the third separated segment is separated into at least two fourth separated segments according to the separation rule of the character string corresponding to it, thereby obtaining a third ASCII character string, which can adjust the word segmentation result and improve the accuracy of word segmentation.

[0175] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the server 400 provided in the embodiment of the present application,

[0176] The adjusting unit 404 is further configured to combine the separated segments in the third ASCII character string according to the second text database to obtain a fourth ASCII character string, wherein the second text database includes at least one custom-obtained character string.

[0177] Secondly, in this embodiment, by setting up a second text database, when there are multiple separated segments in the third ASCII character string that can be combined into a character string in the second text database, these separated segments are recombined into one separated segment, so as to obtain a fourth ASCII character string after adjusting the word segmentation result, which can further adjust the word segmentation result and improve the accuracy of word segmentation.

[0178] Figure 7 7 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 722 (for example, one or more processors) and a memory 732, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 742 or data 744. Among them, the memory 732 and the storage medium 730 may be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 722 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the server 700.

[0179] The server 700 may also include one or more power supplies 726, one or more wired or wireless network interfaces 750, one or more input and output interfaces 758, and / or one or more operating systems 741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0180] The steps performed by the server in the above embodiment can be based on the Figure 7 The server structure shown.

[0181] In the embodiment of the present application, the CPU 722 is used to perform the following steps:

[0182] Obtain a character string to be segmented, wherein the character string to be segmented is an American Standard Code for Information Interchange ASCII character string;

[0183] Performing word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M separated segments, where M is an integer greater than 1;

[0184] The first ASCII character string is segmented by a hidden Markov model to obtain a second ASCII character string, wherein the HMM model is obtained by training with an ASCII character string, and the second ASCII character string includes N separated segments, where N is an integer greater than M.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0190] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A word segmentation method, characterized in that: include: Obtain a character string to be segmented, wherein the character string to be segmented is an American Standard Code for Information Interchange (ASCII) character string, and the character string to be segmented includes an English string, a Pinyin string, or a symbol string; Performing word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M separated segments, where M is an integer greater than 1; The first ASCII character string is segmented by a Hidden Markov Model (HMM) to obtain a second ASCII character string; wherein the HMM model is obtained by training with an ASCII character string, and the second ASCII character string includes N separated segments, where N is an integer greater than M; Get preset adjustment rules; adjusting the word segmentation result of the second ASCII character string according to the preset adjustment rule to obtain a third ASCII character string, wherein the preset adjustment rule is used to combine or re-separate the separated segments in the second ASCII character string; The word segmentation result of the second ASCII character string is adjusted according to the preset adjustment rule to obtain a third ASCII character string, including: If there are multiple continuous first separation segments in the second ASCII character string, combine the multiple continuous first separation segments to obtain a third ASCII character string, wherein the first separation segment contains only a single character; The step of performing word segmentation on the first ASCII character string by using the HMM model to obtain a second ASCII character string includes: Inputting the first ASCII character string into a Hidden Markov Model (HMM) to obtain a state value of each character in the first ASCII character string; Based on the status value of each character in the obtained first ASCII character string, the first ASCII character string is segmented to obtain a second ASCII character string.

2. The word segmentation method according to claim 1, characterized in that: Before performing word segmentation processing on the first ASCII character string through the HMM model to obtain the second ASCII character string, the method further includes: Obtaining first text information; Performing word segmentation processing on the first text information to obtain text information to be processed; Determining target text information according to the text information to be processed, wherein the target text information includes at least one voice element, and each voice element has a corresponding relationship with a character in the text information to be processed, and the voice element is an ASCII character or an ASCII character string; The HMM model is trained by using the target text information to obtain a trained HMM model; The first ASCII character string is segmented by the HMM model to obtain a second ASCII character string, including: The first ASCII character string is segmented by using the trained HMM model to obtain a second ASCII character string.

3. The word segmentation method according to claim 1, characterized in that: Before performing word segmentation processing on the first ASCII character string through the HMM model to obtain the second ASCII character string, the method further includes: Acquire second text information from the first text database, wherein the second text information has a separator, the separator is used to perform word segmentation processing on the second text information, and the second text information is composed of ASCII characters; Training the HMM model using the second text information to obtain a trained HMM model; The first ASCII character string is segmented by the HMM model to obtain a second ASCII character string, including: The first ASCII character string is segmented by using the trained HMM model to obtain a second ASCII character string.

4. The word segmentation method according to claim 1, characterized in that: The word segmentation result of the second ASCII character string is adjusted according to the preset adjustment rule to obtain a third ASCII character string, including: If the second ASCII character string contains a second delimited segment, combining the second delimited segment with the previous delimited segment to obtain a third ASCII character string; The second separation segment includes characters or character strings in the first preset vocabulary.

5. The word segmentation method according to claim 1, characterized in that: The word segmentation result of the second ASCII character string is adjusted according to the preset adjustment rule to obtain a third ASCII character string, including: If the third separation segment exists in the second ASCII character string, the third separation segment is separated into at least two fourth separation segments to obtain a third ASCII character string; The third separated segment includes a character string in the second preset vocabulary.

6. The word segmentation method according to claim 1, characterized in that: After adjusting the word segmentation result of the second ASCII character string according to the preset adjustment rule to obtain a third ASCII character string, the method further includes: The separated segments in the third ASCII character string are combined according to a second text database to obtain a fourth ASCII character string, wherein the second text database includes at least one custom-obtained character string.

7. A server, characterized in that: include: An acquisition unit, used for acquiring a character string to be segmented, wherein the character string to be segmented is an American Standard Code for Information Interchange ASCII character string, and the character string to be segmented includes an English string, a Pinyin string, or a symbol string; A processing unit, configured to perform word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M delimited segments, where M is an integer greater than 1; The processing unit is further configured to input the first ASCII string into a Hidden Markov Model (HMM) to obtain a state value of each character in the first ASCII string; based on the obtained state value of each character in the first ASCII string, segment the first ASCII string to obtain a second ASCII string; wherein the HMM model is obtained by training with an ASCII string, and the second ASCII string includes N separated segments, where N is an integer greater than M; The server also includes an adjustment unit; The acquisition unit is also used to acquire the preset adjustment rules; The adjusting unit is used to adjust the word segmentation result of the second ASCII character string according to the preset adjustment rule to obtain a third ASCII character string, wherein the preset adjustment rule is used to combine or re-separate the separated segments in the second ASCII character string; The adjustment unit is specifically configured to combine multiple continuous first separation segments to obtain a third ASCII character string if there are multiple continuous first separation segments in the second ASCII character string, wherein the first separation segment only contains a single character.

8. The server according to claim 7, characterized in that: Also includes: Training unit; The acquiring unit is further used to acquire the first text information; The processing unit is further used to perform word segmentation processing on the first text information to obtain text information to be processed; The processing unit is further used to determine target text information according to the text information to be processed, wherein the target text information includes at least one voice element, and each voice element has a corresponding relationship with a character in the text information to be processed, and the voice element is an ASCII character or an ASCII character string; A training unit, used for training the HMM model through the target text information to obtain a trained HMM model; The processing unit is further used to perform word segmentation processing on the first ASCII character string through the trained HMM model to obtain a second ASCII character string.

9. The server according to claim 7, characterized in that: The acquiring unit is further used to acquire second text information from the first text database, wherein the second text information has a separator, the separator is used to perform word segmentation processing on the second text information, and the second text information is composed of ASCII characters; The training unit is further used to train the HMM model using the second text information to obtain a trained HMM model; The processing unit is further used to perform word segmentation processing on the first ASCII character string through the trained HMM model to obtain a second ASCII character string.

10. The server according to claim 7, characterized in that: an adjusting unit, specifically configured to combine the second separation segment with a previous separation segment to obtain a third ASCII character string if the second separation segment exists in the second ASCII character string; The second separation segment includes characters or character strings in the first preset vocabulary.

11. The server according to claim 7, characterized in that: an adjusting unit, specifically configured to, if a third separation segment exists in the second ASCII character string, separate the third separation segment into at least two fourth separation segments to obtain a third ASCII character string; The third separated segment includes a character string in the second preset vocabulary.

12. The server according to claim 7, characterized in that: The adjustment unit is further used to combine the separated segments in the third ASCII character string according to a second text database to obtain a fourth ASCII character string, wherein the second text database includes at least one custom-obtained character string.

13. A server, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, comprising the following steps: Obtain a character string to be segmented, wherein the character string to be segmented is an American Standard Code for Information Interchange (ASCII) character string, and the character string to be segmented includes an English string, a Pinyin string, or a symbol string; Performing word segmentation processing on the character string to be segmented by matching a regular expression to obtain a first ASCII character string, wherein the regular expression includes a predefined character string, and the first ASCII character string includes M separated segments, where M is an integer greater than 1; Inputting the first ASCII character string into a Hidden Markov Model (HMM) to obtain a state value of each character in the first ASCII character string; Based on the state value of each character in the obtained first ASCII string, the first ASCII string is segmented to obtain a second ASCII string; wherein the HMM model is obtained by training with the ASCII string, and the second ASCII string includes N separated segments, where N is an integer greater than M; Get preset adjustment rules; adjusting the word segmentation result of the second ASCII character string according to the preset adjustment rule to obtain a third ASCII character string, wherein the preset adjustment rule is used to combine or re-separate the separated segments in the second ASCII character string; The word segmentation result of the second ASCII character string is adjusted according to the preset adjustment rule to obtain a third ASCII character string, including: If there are multiple continuous first separation segments in the second ASCII character string, combine the multiple continuous first separation segments to obtain a third ASCII character string, wherein the first separation segment contains only a single character; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising instructions, characterized in that When the method is executed on a computer or a processor, the computer or the processor is enabled to execute the method according to any one of claims 1 to 6.

Images