Word segmentation processing method and device, server and storage medium

Abstract

The application discloses a word segmentation processing method and device, a server and a storage medium. The word segmentation processing method comprises the following steps: performing word segmentation processing on an original text to be segmented to obtain a first word segmentation result, wherein the first word segmentation result comprises M words; performing cutting processing on the M words according to a first cutting rule to obtain a first cutting result, wherein the first cutting rule comprises the following steps: cutting adjacent same type characters in the i-th word into a single string, and performing single character cutting on the segmented Chinese character string; performing cutting processing on the M words according to a second cutting rule to obtain a second cutting result, wherein the second cutting rule comprises the following steps: cutting adjacent same type characters in the i-th word into a single string, and not performing single character cutting on the segmented Chinese character string; performing sliding window processing on the i-th word to obtain a first word segmentation set of the i-th word; and updating the first word segmentation result by using the first word segmentation set of the i-th word. The method disclosed by the application can reduce the word segmentation cost.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a word segmentation processing method, apparatus, server and storage medium. Background Technology

[0002] In various scenarios of Chinese natural language processing, text segmentation is often required. For example, in order to retrieve matching products from a product database based on the user's search text, both the user's search text and the text of the products in the product database need to be segmented into words for text matching to retrieve the products.

[0003] In industrial text segmentation schemes, machines typically learn from pre-annotated corpora and then segment the original text based on the learned content. However, since the corpora are usually manually annotated, this requires significant human resources, leading to high costs for obtaining training data and consequently, high segmentation costs. Summary of the Invention

[0004] This application provides a word segmentation processing method, apparatus, server, and storage medium, which can solve the problem that the cost of obtaining word segmentation training data is high due to the need to spend a lot of human resources to annotate it, thus leading to high word segmentation costs.

[0005] On the one hand, embodiments of this application provide a word segmentation processing method, including:

[0006] The original text to be segmented is segmented to obtain the first segmentation result, which includes M words. The i-th word in the M words includes Chinese text and / or non-Chinese text, where M is a positive integer.

[0007] The M words are segmented according to the first segmentation rule to obtain the first segmentation result. The first segmentation rule includes: segmenting adjacent characters of the same type in the i-th word into individual strings, and performing single-character segmentation on the segmented Chinese character strings, while not performing single-character segmentation on non-Chinese character strings, i∈[1,M], and i is an integer;

[0008] The M words are segmented according to the second segmentation rule to obtain the second segmentation result. The second segmentation rule includes segmenting adjacent characters of the same type in the i-th word into a single string, and not segmenting Chinese character strings and non-Chinese character strings into single characters.

[0009] If the first segmentation result and the second segmentation result meet the preset conditions, the i-th word is subjected to sliding window processing to obtain the first word segmentation set of the i-th word;

[0010] Update the first word segmentation result using the first word segmentation set of the i-th word.

[0011] On the other hand, embodiments of this application provide a word segmentation processing apparatus, including:

[0012] The first word segmentation module is used to segment the original text to be segmented and obtain the first word segmentation result. The first word segmentation result includes M words, and the i-th word in the M words includes Chinese text and / or non-Chinese text, where M is a positive integer.

[0013] The first segmentation module is used to segment the M words according to the first segmentation rule to obtain the first segmentation result. The first segmentation rule includes: segmenting adjacent characters of the same type in the i-th word into a single string, and performing single-character segmentation on the segmented Chinese character strings, while not performing single-character segmentation on non-Chinese character strings, i∈[1,M], and i is an integer;

[0014] The second segmentation module is used to segment the M words according to the second segmentation rule to obtain the second segmentation result. The second segmentation rule includes segmenting adjacent characters of the same type in the i-th word into a single string, and not performing single-character segmentation on the segmented Chinese character strings and non-Chinese character strings.

[0015] A sliding window module is used to perform sliding window processing on the i-th word when the first segmentation result and the second segmentation result meet preset conditions, so as to obtain the first word segmentation set of the i-th word;

[0016] The first update module is used to update the first word segmentation result using the first word segmentation set of the i-th word.

[0017] In another aspect, embodiments of this application provide a server, the server comprising: a processor and a memory storing computer program instructions;

[0018] When the processor executes the computer program instructions, it implements the word segmentation processing method provided in the first aspect.

[0019] In another aspect, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the word segmentation processing method provided in the first aspect.

[0020] The word segmentation processing method, apparatus, server, and storage medium of this application first perform word segmentation processing on the original text to be segmented, obtaining a first word segmentation result. Then, the words in the first word segmentation result are further processed using two different segmentation methods to obtain a first segmentation result and a second segmentation result. Next, if the first and second segmentation results meet preset conditions, a sliding window processing is applied to the word to obtain a word segmentation set. The first word segmentation result is updated using the word segmentation set. The updated word segmentation result has higher quality and can meet application requirements. Using the embodiments of this application, automatic multi-granularity word segmentation of original text can be achieved. Word segmentation processing no longer relies on a corpus, thereby avoiding the need for significant human resources to obtain training data of different granularities and reducing the cost of text segmentation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating an embodiment of the word segmentation processing method provided in this application.

[0023] Figure 2 This is a flowchart illustrating another embodiment of the word segmentation processing method provided in this application.

[0024] Figure 3 This is a flowchart illustrating another embodiment of the word segmentation processing method provided in this application.

[0025] Figure 4 This is a diagram illustrating the word segmentation results obtained by segmenting the original text of an example using various word segmentation methods.

[0026] Figure 5 This is a diagram illustrating the word segmentation results obtained by segmenting the original text of another example using various word segmentation processing methods.

[0027] Figure 6 This is a schematic diagram of one embodiment of a word segmentation processing device provided in this application.

[0028] Figure 7 This is a schematic diagram of the hardware structure of one embodiment of the server provided in this application. Detailed Implementation

[0029] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0031] To address the problems of the prior art, embodiments of this application provide a word segmentation processing method, apparatus, server, and storage medium. The word segmentation processing method provided in this application embodiment will be described first below.

[0032] Figure 1 This is a flowchart illustrating an embodiment of the word segmentation method provided in this application. Figure 1 As shown, word segmentation methods include:

[0033] S102, perform word segmentation on the original text to be segmented to obtain the first word segmentation result. The first word segmentation result includes M words. The i-th word among the M words includes Chinese text and / or non-Chinese text, where M is a positive integer.

[0034] Non-Chinese text includes, but is not limited to, at least one of the following: Arabic numerals, English letters and symbols. English letters include uppercase and lowercase letters. Symbols may include mathematical symbols (such as “+”, “﹣”, “×”, “÷”, etc.), punctuation marks (such as “;”, “{}”, “《》”, etc.), and unit symbols (such as “℃”, “㎎”, “㎏”, etc.).

[0035] As an example, S102 may specifically include: using a smart segmenter to perform smart segmentation on the original text to be segmented, and obtaining the first segmentation result.

[0036] For example, the original text to be segmented is "The iPhone 12 of Apple X and Huawei Mate are both very good and people can't put them down". Perform smart segmentation on this original text to obtain the first segmentation result. The first segmentation result is specifically "Apple X|mobile phone|iPhone 12|and|Huawei|Mate|both|very good|let|people|can't put them down".

[0037] The word segmentation processing method further includes:

[0038] S104, perform segmentation processing on M words according to the first segmentation rule to obtain the first segmentation result. The first segmentation rule includes: segment adjacent same-type characters in the i-th word into single strings, and perform single-character segmentation on the segmented Chinese character strings, without performing single-character segmentation on non-Chinese character strings, where i ∈ [1, M] and i is an integer.

[0039] Among them, the i-th word can be any one of the M words, and the i-th word can include Chinese characters. When segmenting adjacent same-type characters in the i-th word into single strings, since adjacent same-type characters placed together are usually considered to form a word and have a combined meaning, the adjacent same-type characters in the i-th word are segmented together to form a single string, while non-adjacent characters are separated. Among them, the character types include but are not limited to Chinese characters, letters (regardless of case), and numbers. The segmented strings are divided into Chinese character strings and non-Chinese character strings. A Chinese character string refers to a string whose characters are all Chinese characters, and a non-Chinese character string refers to a string other than a Chinese character string. For example, a non-Chinese character string is a string whose characters are all letters or a string whose characters are all numbers.

[0040] For example, if the i-th word is "Apple X", when segmenting the i-th word "Apple X" according to the first segmentation rule, the first segmentation result is three words. Specifically, the first segmentation result is "Apple|X".

[0041] For another example, if the i-th word is "can't put them down", when segmenting the i-th word according to the first segmentation rule, the first segmentation result is 4 words. Specifically, the first segmentation result is "can't|put|them|down".

[0042] The word segmentation processing method further includes:

[0043] S106, perform segmentation processing on M words according to the second segmentation rule to obtain the second segmentation result. The second segmentation rule includes not performing single-character segmentation on the segmented Chinese character strings and non-Chinese character strings.

[0044] Among them, the difference between the first segmentation rule and the second segmentation rule is that: the first segmentation rule performs single-character segmentation on Chinese character strings, while the second segmentation rule does not perform single-character segmentation on Chinese character strings.

[0045] For example, if the i-th word is "apple x", when segmenting the i-th word "apple x" according to the second segmentation rule, the second segmentation result is two words. Specifically, the second segmentation result is "apple|x".

[0046] For another example, if the i-th word is "can't bear to part with", when segmenting the i-th word according to the second segmentation rule, since all characters in the word "can't bear to part with" are Chinese characters, the second segmentation result is still "can't bear to part with".

[0047] The word segmentation processing method further includes:

[0048] S108, when the first segmentation result and the second segmentation result meet the preset conditions, perform a sliding window process on the i-th word to obtain the first word segmentation set of the i-th word;

[0049] S110, update the first word segmentation result using the first word segmentation set of the i-th word.

[0050] In the embodiments of the present application, first perform word segmentation processing on the original text to be segmented to obtain the first word segmentation result, and perform two types of segmentation processing on the i-th word in the first word segmentation result respectively to obtain the first segmentation result and the second segmentation result. Then, when the first segmentation result and the second segmentation result meet the preset conditions, perform a sliding window process on the i-th word to obtain the first word segmentation set of the i-th word. Update the first word segmentation result using the first word segmentation set of the i-th word. In this way, automatic multi-granularity word segmentation of the original text can be achieved without a corpus, thus avoiding occupying a large amount of human resources to obtain word segmentation training corpora of different granularities and reducing the word segmentation cost of the text.

[0051] The embodiments of the present application can be applied to the scenario of product search. In this scenario, the above-mentioned original text can be the text used to search for products in the product library. For example, the above-mentioned original text can be the search text entered by the user to search for products. The above-mentioned original text can also be the text used to describe the products in the product library.

[0052] For example, first use the word segmentation processing method of the embodiments of the present application to segment the description text of the products in the product library to obtain the word segmentation result of the products. When the user enters the search text, then use the word segmentation processing method of the embodiments of the present application to segment the search text entered by the user to obtain the word segmentation result of the user's search text. Then, match the word segmentation result of the search text with the word segmentation results of each product in the product library respectively, and recall the products that match the user's search text in the product library according to the matching results.

[0053] Of course, the application scenarios of this application embodiment are not limited to product search scenarios, but can also be applied to web search scenarios, multimedia information recommendation and social media data analysis scenarios.

[0054] For example, if the embodiments of this application are applied to a web search scenario, the word segmentation processing method of the embodiments of this application can be used to segment the search text input by the user and the text in the webpage, and the matching webpage can be obtained based on the word segmentation results.

[0055] For example, if the embodiments of this application are applied to a multimedia information recommendation scenario, the word segmentation processing method of the embodiments of this application can be used to segment the relevant text of multimedia information in the multimedia library, and the user's point of interest information can be matched with the word segmentation results of each multimedia information in the multimedia library to match multimedia information that the user may be interested in, and then recommend the matched multimedia information to the user. Multimedia information recommendation includes, but is not limited to, video recommendation (such as short video recommendation) and article recommendation.

[0056] For example, if the embodiments of this application are applied to a scenario of social media data analysis, then the word segmentation processing method of the embodiments of this application can be used to perform text segmentation on the user's social media data, and the user's preferences and behavioral biases can be determined based on the results of text segmentation.

[0057] In one or more embodiments of this application, S108 may specifically include:

[0058] If the number of strings in the first segmentation result is greater than the first preset value (e.g., the first preset value is 1), and the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is greater than the second preset value (e.g., the second preset value is 1), and the length of the i-th word is greater than the third preset value (e.g., the third preset value is 4), then the i-th word is subjected to sliding window processing; where the third preset value is greater than the first preset value.

[0059] Here, the length of the i-th word is the number of words in the first segmentation result of the i-th word. For example, if the i-th word is "apple x", then the length of the i-th word is 3.

[0060] In the example above, the preset condition includes a condition that the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is greater than a second preset value. If the first segmentation result and the second segmentation result satisfy this condition, it means that the i-th word has a relatively large number of Chinese characters. For example, if the second preset value is 1, then if the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is greater than 1, it means that the i-th word has at least 2 Chinese characters.

[0061] In addition, the preset conditions also include that the number of strings in the first segmentation result is greater than the first preset value, and the length of the i-th word is greater than the third preset value. If the first segmentation result and the i-th word also meet this condition, it means that the i-th word is relatively long, so a sliding window process needs to be applied to the i-th word to obtain more segmentation results for the i-th word.

[0062] In one or more embodiments of this application, the sliding window processing includes forward sliding window processing and reverse sliding window processing performed with a single string as the step size; the sliding window processing for the i-th word includes:

[0063] Perform sliding window processing from the second string of the i-th word to the last string;

[0064] And perform sliding window processing from the last string of the i-th word to the first string of the i-th word.

[0065] The following explanation uses "apple x" as an example to illustrate the forward and reverse sliding window processing. Both forward and reverse sliding window processing have a window length of a single string. This single string refers to a string segmented using the first segmentation rule.

[0066] When performing forward sliding window processing on the i-th word, based on the string in the first segmentation result of the i-th word, sliding window processing is performed from the second string to the last string of the i-th word, resulting in the following results: "apple" and "applex".

[0067] When performing reverse sliding window processing on the i-th word, based on the string in the first segmentation result of the i-th word, sliding window processing is performed from the last string of the i-th word to the first string of the i-th word, resulting in the following results: "x", "fruit x", and "apple x".

[0068] In this embodiment of the application, by performing forward sliding window processing and reverse sliding window processing on the i-th word, as many string combinations of the i-th word as possible can be obtained, so that the original text has more and more comprehensive word segmentation results.

[0069] In one or more embodiments of this application, if the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is less than or equal to a second preset value, or if the length of the i-th word is less than or equal to a third preset value, the word segmentation method further includes performing single-character item word segmentation on the i-th word, wherein performing single-character item word segmentation on the i-th word specifically includes:

[0070] Identify the single-character item words in the i-th word;

[0071] Separate the words adjacent to the single-character item words in the i-th word to obtain the first word segmentation set of the i-th word;

[0072] Update the first word segmentation result using the first word segmentation set of the i-th word.

[0073] Among them, single-character item terms are words that represent items and have one Chinese character. Single-character item terms can be single-character words that are commonly used to describe items in a certain scenario. Taking the second-hand e-commerce scenario as an example, single-character item terms include, but are not limited to: "table", "bed", "cabinet", "shoes", and "bag".

[0074] When segmenting single-character item words, in order to avoid ambiguity caused by segmenting the middle word of the i-th word, if the last character of the i-th word is a single-character item word, then the i-th word is segmented as a single-character item word.

[0075] For example, if the i-th word is "single shoulder bag", and the last character "bag" is a single-character item word, then the single-item word segmentation result of the i-th word is "single shoulder|bag".

[0076] In this embodiment, the i-th word is segmented into single-character item words to obtain single-character item words within the i-th word, thus adapting to user habits. Since users prefer to input fewer words in their searches, such as "table" or "bag," single-character item word segmentation is performed to cater to user habits. Developers can adjust this based on user habits in their specific scenarios.

[0077] It should be noted that if the i-th word does not contain any single-character item terms, then no processing is required for the i-th word as a single-character item term. For example, if the i-th word is "apple x", then no processing is performed on this word since it does not contain any single-character item terms.

[0078] In one or more embodiments of this application, when the original text is text used to search for products in a product database, updating the first word segmentation result using the first word segmentation set of the i-th word includes:

[0079] Replace the i-th word in the first word segmentation result with the first word segmentation set of the i-th word.

[0080] Since the first segmentation set obtained from the new segmentation of the i-th word includes more fine-grained segments of the i-th word, replacing the i-th word in the first segmentation result with words from the first segmentation set can make the first segmentation result biased towards retaining fine-grained segments, minimizing long words, and thus increasing the matching degree between the original text and the text in the text library. For example, when the original text is a search term entered by the user, it can increase the matching degree between the search term and the product text.

[0081] In one or more embodiments of this application, when the original text is text used to search for products in a product database, the word segmentation method may further include:

[0082] Replace the i-th word in the first segmentation result with a word from the second segmentation result.

[0083] Since the second segmentation result includes more fine-grained word segments of the i-th word, replacing the i-th word in the first segmentation result with a word in the second segmentation result can make the first segmentation result more inclined to retain fine-grained word segments, minimize long words, and thus increase the matching degree between the original text and the text in the text library.

[0084] In one or more embodiments of this application, when the original text is text used to describe products in a product library, updating the first word segmentation result using the first word segmentation set of the i-th word includes:

[0085] Add the first segmentation set of the i-th word to the first segmentation result.

[0086] By retaining the i-th word in the first word segmentation result, the first word segmentation result contains not only the long word (i.e., the i-th word) obtained from the original text segmentation, but also the short word obtained from the i-th word segmentation. Therefore, the first word segmentation result includes more possible words, which can increase the matching degree between the original text and the text in the text library.

[0087] In one or more embodiments of this application, when the original text is text used to describe products in a product library, the word segmentation method may further include:

[0088] Add the words from the second segmentation result to the first segmentation result.

[0089] This allows the first word segmentation result to include more possible words, increasing the matching degree between the search text and the product text.

[0090] In one or more embodiments of this application, the word segmentation processing method further includes:

[0091] The words in the first word segmentation set of the i-th word obtained by sliding window processing are segmented to obtain the second word segmentation set of the i-th word;

[0092] The words in the second segmentation set of the i-th word are merged into the first segmentation result and deduplication is performed.

[0093] In one or more embodiments of this application, the word segmentation processing method further includes:

[0094] If any k-th word in the second word segmentation set is a Chinese word with a length greater than the preset word length, for example, if the preset word length is 1 and the k-th word is a Chinese word with a length of 2 or 3, then the k-th word is segmented into a single-character item word to obtain the single-character item word segmentation result of the k-th word, where k is a positive integer;

[0095] Update the first word segmentation result using the single-character item word segmentation result of the k-th word.

[0096] Regarding the segmentation of the k-th word as a single-character item word, since the segmentation of the k-th word in the second word segmentation set as a single-character item word is similar to the segmentation of the i-th word as described above, and the segmentation of the i-th word as a single-character item word has already been explained above, the segmentation of the k-th word as a single-character item word will not be repeated here.

[0097] Updating the first word segmentation result using the single-character item word segmentation result of the k-th word can be implemented in the following two cases:

[0098] Scenario 1: The original text is the text used to search for products in the product database.

[0099] Updating the first word segmentation result using the single-character item word segmentation result of the kth word specifically includes: replacing the kth word in the first word segmentation result with the single-character item word segmentation result of the kth word.

[0100] Scenario 2: The original text is used to describe the products in the product database.

[0101] Updating the first word segmentation result using the single-character item word segmentation result of the k-th word specifically includes: adding the single-character item word segmentation result of the k-th word to the first word segmentation result.

[0102] In one or more embodiments of this application, the word segmentation processing method further includes:

[0103] If the length of the i-th word is greater than the fourth preset value, the i-th word is re-segmented to obtain the third segmentation set, and the third segmentation set is the new segmentation result of the i-th word;

[0104] Replace the i-th word in the first word segmentation result with the third word segmentation set.

[0105] For example, after S102 and before S104, the i-th word is re-segmented. In this way, long words in the first segmentation result are re-segmented, which can reduce the difficulty of matching long words.

[0106] In one or more embodiments of this application, the word segmentation processing method further includes:

[0107] If the i-th word is re-segmented and no new segmentation result is obtained for the i-th word, then the single-character item words in the i-th word are determined.

[0108] Separate the words adjacent to the single-character item words in the i-th word to obtain the fourth word set of the i-th word;

[0109] Replace the i-th word in the first word segmentation result with the fourth word segmentation set.

[0110] To avoid ambiguity caused by segmenting the middle word of the i-th word, when the single-character item word in the i-th word is the last character of the i-th word, the character adjacent to the single-character item word in the i-th word is separated from the single-character item word.

[0111] The following describes the embodiments of this application in the context of product search.

[0112] Figure 2 This is a flowchart illustrating another embodiment of the word segmentation method provided in this application. Figure 2 As shown, word segmentation methods include:

[0113] S202, Obtain the original text to be segmented. The original text can be the text used to search for products in the product database.

[0114] S204, perform word segmentation on the original text to obtain the first word segmentation result, which is represented by res, where res = [word1, word2...wordM], and M represents the total number of words in the first word segmentation result;

[0115] S206, determine whether Len(wordi) is greater than L, where Len(wordi) represents the length of the i-th word in the first word segmentation result, and wordi can be any word in the first word segmentation result. If Len(wordi) is greater than L, execute S208; if Len(wordi) is not greater than L, execute S228. Here, i is a positive integer, and L is an integer greater than 1, such as L = 4.

[0116] S208, perform word segmentation on wordi;

[0117] S210, determine if wordi has new word segmentation results. If the result is yes, execute S214; if the result is no, execute S212.

[0118] S212. Perform single-item word segmentation on wordi to obtain the single-item word segmentation result of wordi. Among them, the set of words in the single-item word segmentation result is the above-mentioned fourth segmentation set;

[0119] S214. Replace the original word wordi with the new segmentation result. Specifically, when there is a new segmentation result for wordi, replace the original word wordi with the new segmentation result of wordi. When there is no new segmentation result for wordi, replace the original word wordi with the fourth segmentation set. In this way, the updated wordi is obtained;

[0120] S216. Segment wordi according to the first segmentation rule and the second segmentation rule respectively to obtain the first segmentation result Result1 and the second segmentation result Result2. Among them, wordi in S216 is any one of the words in the updated wordi in S214. For example, in S214, if the original word wordi is "Braised beef noodles", and the words "Braised" and "beef noodles" segmented in S208 are used to replace the original word wordi, then wordi in S216 is "Braised" or "beef noodles";

[0121] S218. Judge whether Len(Result1) is greater than 1. Len(Result1) represents the number of strings in the first segmentation result Result1. When Len(Result1) is greater than 1, execute S220. When Len(Result1) is not greater than 1, do nothing;

[0122] S220. Judge whether the difference obtained by subtracting Len(Result2) from Len(Result1) is greater than 1. When the judgment result is yes, execute S228. When the judgment result is no, execute S222. Among them, Len(Result2) represents the number of strings in the second segmentation result Result2;

[0123] S222. Judge whether the conditions of Len(wordi) = 2 and wordi being a Chinese word are satisfied. When the judgment result is yes, execute S224. When the judgment result is no, do nothing;

[0124] S224. Perform single-item word segmentation on wordi;

[0125] S226. Replace the word wordi before segmentation with the single-item word segmentation result of wordi;

[0126] S228: Determine whether Len(wordi) is greater than or equal to M, Len(wordi) is less than or equal to L, and wordi is a Chinese word. If the result is yes, execute S230; if the result is no, execute S234.

[0127] S230, perform single-item word segmentation on wordi;

[0128] S232, Replace the word wordi before segmentation with the single-item word segmentation result of wordi;

[0129] S234, perform sliding window processing on wordi to obtain the first word segmentation set. The sliding window processing includes forward sliding window processing and reverse sliding window processing. The forward sliding window processing obtains the word forwardSlidStr=Result1[1,bi],(bi=2,…,len(Result1)), and the reverse sliding window processing obtains the word backwardSlidStr=Result1[fi,len(Result1)],(fi=1,…,len(Result1)). The words obtained by the forward sliding window processing and the words obtained by the reverse sliding window processing form the first word segmentation set.

[0130] S236, Segment the words in the first segmentation set to obtain the second segmentation set. The words in the second segmentation set are represented by SlidStr_word;

[0131] S238, determine whether SlidStr_word is a Chinese word, and whether len(SlidStr_word) is equal to 2 or 3. If SlidStr_word is a Chinese word and len(SlidStr_word) is equal to 2 or 3, execute S240. If SlidStr_word is not a Chinese word, and / or len(SlidStr_word) is neither equal to 2 nor equal to 3, execute S244. Here, len(SlidStr_word) represents the length of SlidStr_word.

[0132] S240, perform single-item word segmentation on SlidStr_word;

[0133] S242, replace the original word SlidStr_word with the single-item word segmentation result of SlidStr_word to update the words in the first word segmentation set obtained by sliding window processing;

[0134] S244: Remove duplicate words from the first word segmentation set obtained by sliding window processing and replace the original words;

[0135] S246, aggregate the word segmentation results and update the first word segmentation result of the original text.

[0136] Figure 3 This is a flowchart illustrating another embodiment of the word segmentation processing method provided in this application. For example... Figure 3 As shown, word segmentation methods include:

[0137] S302, Obtain the original text to be segmented. The original text can be the text used to search for products in the product database.

[0138] S304, perform word segmentation on the original text to obtain the first word segmentation result, which is represented by res, where res = [word1, word2...wordM], and M represents the total number of words in the first word segmentation result;

[0139] S306, determine whether Len(wordi) is greater than L, where Len(wordi) represents the length of the i-th word in the first word segmentation result, and wordi can be any word in the first word segmentation result. If Len(wordi) is greater than L, execute S308; if Len(wordi) is not greater than L, execute S328. Here, i is a positive integer, and L is an integer greater than 1, such as L = 4.

[0140] S308, perform word segmentation on wordi;

[0141] S310, determine if wordi has new word segmentation results. If the result is yes, execute S314; if the result is no, execute S312.

[0142] S312, perform single-item word segmentation on wordi to obtain the single-item word segmentation result of wordi, wherein the set of words in the single-item word segmentation result is the fourth word segmentation set mentioned above.

[0143] S314, replace the original word i with the new word segmentation result. Specifically, if there is a new word segmentation result for word i, replace the original word i with the new word segmentation result for word i. If there is no new word segmentation result for word i, replace the original word i with the fourth word segmentation set. In this way, the updated word i is obtained.

[0144] S316, segment wordi according to the first segmentation rule and the second segmentation rule respectively to obtain the first segmentation result Result1 and the second segmentation result Result2. Here, wordi in S316 is any one of the updated wordi in S314. For example, in S314, the original wordi is "Braised Beef Noodles", and if we replace the original wordi with the words "Braised" and "Beef Noodles" segmented in S308, then wordi in S316 is "Braised" or "Beef Noodles".

[0145] S318, determine whether Len(Result1) is greater than 1. Len(Result1) represents the number of strings in the first segmentation result Result1. If Len(Result1) is greater than 1, execute S320; if Len(Result1) is not greater than 1, do nothing.

[0146] S320, determine whether Len(Result2) is greater than 2 and whether the difference obtained by subtracting Len(Result2) from Len(Result1) is greater than 1. If the judgment result is yes, execute S328; if the judgment result is no, execute S322. Here, Len(Result2) represents the number of strings in the second segmentation result Result2.

[0147] S322, determine whether the conditions of Len(wordi) = 2 and wordi being a Chinese word are met. If the judgment result is yes, execute S324; if the judgment result is no, do nothing.

[0148] S324, perform single-item word segmentation on wordi.

[0149] S326, retain the word wordi before segmentation. That is, in S346, when aggregating the segmentation results, the aggregated segmentation results include the single-item word segmentation result of wordi and the word wordi before segmentation.

[0150] S328, determine whether Len(wordi) is greater than or equal to M, Len(wordi) is less than or equal to L, and wordi is a Chinese word. If the judgment result is yes, execute S330; if the judgment result is no, execute S334.

[0151] S330, perform single-item word segmentation on wordi.

[0152] S332, retain the word wordi before segmentation. That is, in S346, when aggregating the segmentation results, the aggregated segmentation results include the single-item word segmentation result of wordi and the word wordi before segmentation.

[0153] S334, perform sliding window processing on wordi to obtain the first word segmentation set. The sliding window processing includes forward sliding window processing and reverse sliding window processing. The forward sliding window processing obtains the word forwardSlidStr = Result1[1,bi], (bi = 2, ..., len(Result1)). The reverse sliding window processing obtains the word backwardSlidStr = Result1[fi, len(Result1)], (fi = 1, ..., len(Result1)). The words obtained by the forward sliding window processing and the words obtained by the reverse sliding window processing form the first word segmentation set.

[0154] S336, Segment the words in the first segmentation set to obtain the second segmentation set. The words in the second segmentation set are represented by SlidStr_word;

[0155] S338, determine whether SlidStr_word is a Chinese word, and whether len(SlidStr_word) is equal to 2 or 3. If SlidStr_word is a Chinese word and len(SlidStr_word) is equal to 2 or 3, execute S340. If SlidStr_word is not a Chinese word, and / or len(SlidStr_word) is neither equal to 2 nor equal to 3, execute S344. Here, len(SlidStr_word) represents the length of SlidStr_word.

[0156] S340, perform single-item word segmentation on SlidStr_word;

[0157] S342, replace the original word SlidStr_word with the single-item word segmentation result of SlidStr_word to update the words in the first word segmentation set obtained by sliding window processing;

[0158] S344: Remove duplicate words from the first word segmentation set obtained by sliding window processing and retain the original word SlidStr_word;

[0159] S346, aggregate the word segmentation results and update the first word segmentation result of the original text.

[0160] It should be noted that, Figure 2 and Figure 3 The main difference is:

[0161] ●In Figure 2 In the process, after segmenting wordi according to the second segmentation rule to obtain the second segmentation result Result2, the second segmentation result Result2 is used to replace wordi in the original word segmentation result; while Figure 3 In the process, after segmenting wordi according to the second segmentation rule to obtain the second segmentation result Result2, the second segmentation result Result2 is added to the first word segmentation result, and wordi in the first word segmentation result is retained.

[0162] ●In Figure 2 In the code, S226, S232, and S244 all involve word replacement, that is, replacing wordi in the first word segmentation result with words obtained from single-item word segmentation or sliding window processing. Figure 3 In the first segmentation result, S326, S332 and S344 all perform new word retention, that is, not only do the words obtained by single item word segmentation and the words obtained by sliding window processing be included in the first segmentation result, but the original word wordi is also retained in the first segmentation result.

[0163] ●In Figure 2 In step S220, it checks whether the difference between Len(Result1) and Len(Result2) is greater than 1. Figure 3 In S320, it not only determines whether the difference between Len(Result1) and Len(Result2) is greater than 1, but also whether Len(Result2) is greater than 2.

[0164] The following is through Figure 4 and Figure 5 The examples further illustrate the word segmentation processing method provided in the embodiments of this application.

[0165] For example, such as Figure 4 As shown, assuming the original text is the search text input by the user to search for products in the product database, and the search text is "vivoy3", then the word segmentation result obtained by using the iksmart model to segment the search text is "vivoy3", the word segmentation result obtained by using the ikmax model to segment the search text is "vivoy3|vivoy|vivo|vo|3", the word segmentation result obtained by using the newsmart model to segment the search text is "vivo|y3", and the word segmentation result obtained by using the newmax model is "vivo|y|3".

[0166] Among them, iksmart, ikmax and newsmart are three word segmentation models in related technologies, while the newmax model is a word segmentation model implemented using the word segmentation processing method provided in this application. In other words, the newmax model is the word segmentation model provided in this application.

[0167] Assume that the original text is the theme of a product in the product library, and the original text is "90% new vivo vivo y3 64g vermilion". Then, the word segmentation result obtained by using the ikmax model for the original text is "9|成|新|vivo|vo|vivo|vo|y3|y|3|64g|64|g|绯|玉|红", and the word segmentation result obtained by using the newmax model for the original text is "9成新|vivo|vivo|y3|64g|绯玉红|9|成新|y|3|64|g|绯玉|红|绯|玉红".

[0168] Assume that the original text is the description content of a product in the product library, and the original text is "90% new vivo vivo y364g vermilion". Then, the word segmentation result obtained by using the iksmart model for the original text is "9成|新|vivo|y3|64g|绯玉红", and the word segmentation result obtained by using the newsmart model for the original text is "9成新|vivo|vivo|y3|64g|绯玉红".

[0169] For another example, as Figure 5 shown, assume that the original text is the search text entered by the user to search for products in the product library, and the search text is "large plush toy". Then, the word segmentation result obtained by using the iksmart model for the search text is "大|毛绒玩具", and the word segmentation result obtained by using the ikmax model for the search text is "大|毛绒玩具|毛绒|玩具", and the word segmentation result obtained by using the newsmart model for the search text is "大|毛绒|玩具", and the word segmentation result obtained by using the newmax model for the search text is "大|毛绒|玩具".

[0170] Assume that the original text is the theme of a product in the product library, and the original text is "brand new large plush toys alpaca big bear Mickey Mouse Minnie Mouse birthday gift present". Then, the word segmentation result obtained by using the ikmax model for the original text is "全新|新大|大型|毛绒玩具|毛绒|玩具|羊|驼|大熊|米奇|米老鼠|生日礼物|生日|礼物|礼品", and the word segmentation result obtained by using the newmax model for the original text is "全新|大型|毛绒|玩具|羊驼|大熊|米奇|米老鼠|生日|礼物|礼品|大|型|熊|老|鼠|老鼠|米|米老".

[0171] Assuming the original text is a description of a product in a product database, and the original text is "Brand new large plush toy alpaca bear Mickey Mouse birthday gift, good quality, good feel, super cute", then the word segmentation result obtained by using the iksmart model to segment the original text is "brand new|large|plusl toy|alpaca|bear|Mickey|Mickey Mouse|birthday gift|gift|quality|good|furry|good feel|good|super|cute", and the word segmentation result obtained by using the newsmart model is "brand new|large|plusl|toy|alpaca|bear|Mickey|Mickey Mouse|birthday gift|gift|quality|good|furry|good|feel|good|super|cute".

[0172] from Figure 5 It can be seen that when segmenting product topics using the IKMAX and NewMAX models, the IKMAX model yields 15 words, while the NewMAX model (the segmentation model provided in this application) yields 19 words. Therefore, the NewMAX model (the segmentation model provided in this application) includes more granular segmentation results, resulting in higher search recall. Meanwhile, Newsmart ensures search recall accuracy, thus comprehensively improving both the recall and accuracy of search text matching.

[0173] Corresponding to the word segmentation method provided in this application, this application also provides a word segmentation apparatus. Figure 6 This is a schematic diagram of the structure of one embodiment of a word segmentation processing device provided in this application. Figure 6 As shown, the word segmentation processing device 400 includes:

[0174] The first word segmentation module 402 is used to segment the original text to be segmented and obtain the first word segmentation result. The first word segmentation result includes M words, and the i-th word among the M words includes Chinese text and / or non-Chinese text, where M is a positive integer.

[0175] The first segmentation module 404 is used to segment M words according to the first segmentation rule to obtain the first segmentation result. The first segmentation rule includes: segmenting adjacent characters of the same type in the i-th word into a single string, and performing single-character segmentation on the segmented Chinese character string, while not performing single-character segmentation on non-Chinese character strings, i∈[1,M], and i is an integer;

[0176] The second segmentation module 406 is used to segment M words according to the second segmentation rule to obtain the second segmentation result. The second segmentation rule includes segmenting adjacent characters of the same type in the i-th word into a single string, and not performing single-character segmentation on the segmented Chinese character string and non-Chinese character string.

[0177] The sliding window module 408 is used to perform sliding window processing on the i-th word when the first segmentation result and the second segmentation result meet the preset conditions, so as to obtain the first word set of the i-th word;

[0178] The first update module 410 is used to update the first word segmentation result using the first word segmentation set of the i-th word.

[0179] In this embodiment, the original text to be segmented is first processed to obtain a first segmentation result. Then, the i-th word in the first segmentation result is subjected to two different segmentation processes to obtain a first segmentation result and a second segmentation result. Next, a sliding window process is applied to the i-th word based on the first and second segmentation results to obtain a first segmentation set for the i-th word. The first segmentation result is updated using this first segmentation set. This allows for automatic multi-granularity segmentation of the original text without the need for a corpus, thus avoiding the need for extensive human resources for annotation and reducing the cost of text segmentation.

[0180] In one or more embodiments of this application, the sliding window module 408 is specifically used for:

[0181] If the number of strings in the first segmentation result is greater than the first preset value, and the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is greater than the second preset value, and the length of the i-th word is greater than the third preset value, then the i-th word is subjected to sliding window processing; wherein, the third preset value is greater than the first preset value.

[0182] In one or more embodiments of this application, the sliding window processing includes forward sliding window processing and reverse sliding window processing performed with a single string as the step size; the sliding window module 408 includes:

[0183] The first sliding window unit is used to perform sliding window processing from the second string of the i-th word to the last string;

[0184] The second sliding window unit is used to perform sliding window processing from the last string of the i-th word to the first string of the i-th word.

[0185] In one or more embodiments of this application, if the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is less than or equal to a second preset value, or if the length of the i-th word is less than or equal to a third preset value, the word segmentation processing device 400 further includes:

[0186] The first determining module is used to determine the single-character item words in the i-th word;

[0187] The third segmentation module is used to separate the characters adjacent to the single-character item words in the i-th word from the single-character item words, and obtain the first word segmentation set of the i-th word;

[0188] The second update module is used to update the first word segmentation result using the first word segmentation set of the i-th word.

[0189] In one or more embodiments of this application, the original text is text used to search for products in a product library.

[0190] In one or more embodiments of this application, the second update module is specifically used for:

[0191] Replace the i-th word in the first word segmentation result with the first word segmentation set of the i-th word.

[0192] In one or more embodiments of this application, the word segmentation processing device 400 further includes:

[0193] The first replacement module is used to replace the i-th word in the first word segmentation result with a word in the second segmentation result.

[0194] In one or more embodiments of this application, the original text is text used to describe the goods in the goods library.

[0195] In one or more embodiments of this application, the second update module is specifically used for:

[0196] Add the first segmentation set of the i-th word to the first segmentation result.

[0197] In one or more embodiments of this application, the word segmentation processing device 400 further includes:

[0198] The add module is used to add words from the second segmentation result to the first segmentation result.

[0199] In one or more embodiments of this application, the word segmentation processing device 400 further includes:

[0200] The second word segmentation module is used to segment the words in the first word segmentation set of the i-th word obtained by sliding window processing, so as to obtain the second word segmentation set of the i-th word.

[0201] The merge and deduplication module is used to merge the words in the second word segmentation set of the i-th word into the first word segmentation result and perform deduplication.

[0202] In one or more embodiments of this application, the word segmentation processing device 400 further includes:

[0203] The third word segmentation module is used to re-segment the i-th word when the length of the i-th word is greater than the fourth preset value, so as to obtain the third word segmentation set, and the third word segmentation set is the new word segmentation result of the i-th word;

[0204] The second replacement module is used to replace the i-th word in the first word segmentation result with the third word segmentation set.

[0205] In one or more embodiments of this application, the word segmentation processing device 400 further includes:

[0206] The second determining module is used to determine the single-character item words in the i-th word when the i-th word is re-segmented and no new segmentation result is obtained for the i-th word.

[0207] The fourth segmentation module is used to separate the characters adjacent to the single-character item words in the i-th word from the single-character item words, so as to obtain the fourth segmentation set of the i-th word;

[0208] The third replacement module is used to replace the i-th word in the first word segmentation result with the fourth word segmentation set.

[0209] This application also provides a server, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the word segmentation processing method as described in any of the above embodiments.

[0210] Figure 7 This is a schematic diagram of the hardware structure of one embodiment of the server provided in this application.

[0211] like Figure 7 As shown, the server may include a processor 501 and a memory 502 storing computer program instructions.

[0212] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0213] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0214] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0215] The processor 501 implements any of the word segmentation processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0216] In one example, the server may also include a communication interface 503 and a bus 510. Wherein, as... Figure 7 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0217] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0218] Bus 510 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0219] Furthermore, in conjunction with the word segmentation processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the word segmentation processing methods in the above embodiments.

[0220] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0221] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Machine-readable media can include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, and can also include radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0222] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0223] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0224] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A word segmentation processing method, characterized by, The method comprises the following steps: performing word segmentation on an original text to be segmented to obtain a first word segmentation result, the first word segmentation result comprising M words, each of the M words comprising Chinese character text and / or non-Chinese character text, M being a positive integer; performing segmentation on the M words according to a first segmentation rule to obtain a first segmentation result, the first segmentation rule comprising: segmenting adjacent characters of the same type in each of the M words into a single string, and performing single-character segmentation on the segmented Chinese character string, and not performing single-character segmentation on the non-Chinese character string, i ∈ [1, M], and i being an integer; performing segmentation on the M words according to a second segmentation rule to obtain a second segmentation result, the second segmentation rule comprising: segmenting adjacent characters of the same type in each of the M words into a single string, and not performing single-character segmentation on the segmented Chinese character string and non-Chinese character string; performing sliding window processing on the i-th word under the condition that the first segmentation result and the second segmentation result meet a preset condition to obtain a first word segmentation set of the i-th word; updating the first word segmentation result using the first word segmentation set of the i-th word; performing sliding window processing on the i-th word under the condition that the first segmentation result and the second segmentation result meet a preset condition, comprising: if the number of strings in the first segmentation result is greater than a first preset value, and the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is greater than a second preset value, and the length of the i-th word is greater than a third preset value, then performing sliding window processing on the i-th word; wherein the third preset value is greater than the first preset value; wherein the length of the i-th word is the number of words in the first segmentation result of the i-th word; if the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is less than or equal to the second preset value, or the length of the i-th word is less than or equal to the third preset value, then determining a single-character item word in the i-th word; cutting the single-character item word from the adjacent characters in the i-th word to obtain a first word segmentation set of the i-th word; updating the first word segmentation result using the first word segmentation set of the i-th word; the single-character item word is a word used to represent an item and having one Chinese character, and the single-character item word cutting is performed only when the single-character item word is the last character in the i-th word.

2. The word segmentation processing method of claim 1, wherein, The sliding window processing comprises forward sliding window processing and reverse sliding window processing performed respectively with a single string as a step; performing sliding window processing on the i-th word, comprising: performing sliding window processing from the second string to the last string of the i-th word; and performing sliding window processing from the last string of the i-th word to the first string of the i-th word.

3. The word segmentation processing method of any one of claims 1-2, wherein, The original text is a text used to search for goods in a goods library.

4. The word segmentation processing method of claim 3, wherein, The updating of the first word segmentation result using the first word segmentation set of the i-th word comprises: replacing the i-th word in the first word segmentation result with the first word segmentation set of the i-th word.

5. The word segmentation method of claim 3, wherein, Further comprising: replace the i-th word in the first segmentation result with a word in the second segmentation result.

6. The word segmentation method of any one of claims 1-2, wherein, The original text is a text for describing goods in a goods library.

7. The word breaking method of claim 6, wherein, The first segmentation result is updated using the first segmentation set of the i-th word, including: The first segmentation set of the i-th word is added to the first segmentation result.

8. The word segmentation method of claim 6, wherein, Further comprising: The word in the first segmentation set of the i-th word obtained by the sliding window processing is segmented to obtain a second segmentation set of the i-th word.

9. The word segmentation method of any one of claims 1-2, wherein, The words in the second segmentation set of the i-th word are merged into the first segmentation result and de-duplicated. Further comprising: In the case where the length of the i-th word is greater than a fourth preset value, the i-th word is segmented again to obtain a third segmentation set, and the third segmentation set is a new segmentation result of the i-th word; 10. The word segmentation method of any one of claims 1-2, wherein, The i-th word in the first segmentation result is replaced by the third segmentation set. Further comprising: In the case where the i-th word is segmented again and no new segmentation result of the i-th word is obtained, a single-character item word in the i-th word is determined; 11. The word breaking method of claim 10, wherein, The character adjacent to the single-character item word in the i-th word is cut off from the single-character item word to obtain a fourth segmentation set of the i-th word; The i-th word in the first segmentation result is replaced by the fourth segmentation set. Comprising: A first segmentation module is configured to segment an original text to be segmented to obtain a first segmentation result, the first segmentation result including M words, an i-th word in the M words including Chinese character text and / or non-Chinese character text, and M being a positive integer; 12. A word segmentation processing apparatus characterized by comprising: A first segmentation module is configured to segment the M words according to a first segmentation rule to obtain a first segmentation result, the first segmentation rule including: cutting adjacent characters of the same type in the i-th word into a single string, and performing single-character segmentation on the segmented Chinese character string, and the non-Chinese character string is not subjected to single-character segmentation, i∈[1,M], and i is an integer; A second segmentation module is configured to segment the M words according to a second segmentation rule to obtain a second segmentation result, the second segmentation rule including cutting adjacent characters of the same type in the i-th word into a single string, and not performing single-character segmentation on the segmented Chinese character string and non-Chinese character string; A sliding window module is configured to perform sliding window processing on the i-th word in the case where the first segmentation result and the second segmentation result meet a preset condition to obtain a first segmentation set of the i-th word; A first update module is configured to update the first segmentation result using the first segmentation set of the i-th word. ​ ​ The sliding window module is configured to perform sliding window processing on the i-th word if the number of strings in the first segmentation result is greater than a first preset value, and a difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is greater than a second preset value, and a length of the i-th word is greater than a third preset value; the third preset value is greater than the first preset value; the length of the i-th word is a number of words in the first segmentation result of the i-th word; The first determination module is configured to determine a single-character item word in the i-th word if the difference between the number of strings in the first segmentation result and the number of strings in the second segmentation result is less than or equal to the second preset value, or the length of the i-th word is less than or equal to the third preset value; the third segmentation module is configured to segment a character adjacent to the single-character item word in the i-th word from the single-character item word to obtain a first segmentation set of the i-th word; and the second update module is configured to update the first segmentation result by using the first segmentation set of the i-th word; the single-character item word is a word used to represent an item and having one Chinese character, and the single-character item word segmentation is performed only when the single-character item word is the last character of the i-th word.

13. A server, characterized by The server comprises a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the word segmentation processing method according to any one of claims 1-11.

14. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the word segmentation processing method according to any one of claims 1-11.

Images