Text processing method and device, electronic equipment and computer readable storage medium

Abstract

The present disclosure provides a text processing method and device, an electronic device and a computer readable storage medium. The method comprises: obtaining a to-be-processed text, and determining a knowledge field to which the to-be-processed text belongs; obtaining a directed acyclic graph corresponding to the knowledge field to which the to-be-processed text belongs; performing word segmentation processing on the to-be-processed text based on the directed acyclic graph and the text length of the to-be-processed text to obtain a word segmentation result corresponding to the to-be-processed text; wherein the directed acyclic graph corresponding to the knowledge field to which the to-be-processed text belongs is created in advance according to the number of times each word in the field text corpus corresponding to the knowledge field to which the to-be-processed text belongs and each word group composed of each word and adjacent words appear in the field text corpus. According to the technical scheme of the present disclosure, the efficiency and accuracy of text word segmentation processing can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and in particular to a text processing method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Natural Language Processing (NLP) requires computers to accurately extract the meaning of text sentences. This process involves syntax, semantic components, semantic structure, and context. All of this is based on accurately segmenting a text sentence into a combination of multiple words; simply put, it's about word segmentation. In deeper NLP processes, such as personalized recommendations, sentiment analysis, topic classification, and public opinion analysis, high-accuracy word segmentation is a prerequisite. However, the emergence of new words often interferes with existing word segmentation software, leading to unsatisfactory segmentation results and consequently affecting subsequent text processing. Summary of the Invention

[0003] This disclosure provides a text processing method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, this disclosure provides a text processing method, which includes: Obtain the text to be processed and determine the knowledge domain to which the text to be processed belongs; Obtain the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs; Based on the directed acyclic graph and the text length of the text to be processed, the text to be processed is segmented into words to obtain the segmentation result corresponding to the text to be processed. The directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is created in advance based on the frequency of each character and the word group formed by each character and its adjacent characters in the domain text corpus to which the text to be processed belongs.

[0005] Secondly, this disclosure provides a text processing apparatus, which includes: The acquisition module is used to acquire the text to be processed and determine the knowledge domain to which the text to be processed belongs; and to acquire the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs. The word segmentation module is used to perform word segmentation on the text to be processed based on the directed acyclic graph and the text length of the text to be processed, so as to obtain the word segmentation result corresponding to the text to be processed. The directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is created in advance based on the number of times each character and each word group composed of each character and its adjacent characters appear in the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs.

[0006] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the text processing method described above.

[0007] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described text processing method.

[0008] According to the technical solutions of the text processing method, apparatus, electronic device, and computer-readable storage medium provided in this disclosure, a directed acyclic graph (DAG) of each character in the domain text corpus is created by statistically analyzing the frequency of occurrence of each character and any two adjacent character phrases in the domain text corpus. During text segmentation, the DAG allows for quick lookup of the frequency of related character combinations, enabling fast and accurate text segmentation based on the statistically obtained frequencies, thus effectively improving the efficiency and accuracy of text segmentation. Furthermore, in the text processing method of this disclosure, both the text and the DAG are divided according to knowledge domains, eliminating the need for manual annotation or dictionary construction for text segmentation, and effectively identifying specialized vocabulary within the knowledge domain of the text.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which: Figure 1 A flowchart of a text processing method provided in this disclosure embodiment; Figure 2 A flowchart of another text processing method provided in this disclosure embodiment; Figure 3 This is a schematic diagram of a directed acyclic graph provided in an embodiment of the present disclosure; Figure 4 for Figure 1 A flowchart of a specific implementation of step S13; Figure 5 for Figure 4 A flowchart of a specific implementation of step S131b; Figure 6 for Figure 1 A flowchart of a specific implementation of step S13; Figure 7 for Figure 1 A flowchart of a specific implementation of step S13; Figure 8 for Figure 7 A flowchart of a specific implementation of step S133d; Figure 9 A block diagram of a text processing apparatus provided in an embodiment of this disclosure; Figure 10 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0012] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0013] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0014] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0015] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0016] According to the text processing method of this disclosure, the text processing method is used to perform text segmentation processing on text in various knowledge domains. It can be applied to any scenario that requires text segmentation processing. Accordingly, the text processing method can be applied to products corresponding to these scenarios, such as intent classification systems, search systems, personalized recommendation systems, sentiment analysis systems, topic classification systems, public opinion analysis systems, etc.

[0017] According to the technical solution of the text processing method provided in this disclosure, by statistically analyzing the frequency of occurrence of each character and any two adjacent character phrases in the domain text corpus, a directed acyclic graph (DAG) is created for each character in the domain text corpus. During text segmentation, the DAG allows for quick querying of the frequency of related character combinations, thus enabling fast and accurate text segmentation and effectively improving the efficiency and accuracy of text segmentation. Furthermore, in the text processing method of this disclosure, both the text and the DAG are divided according to the domain, eliminating the need for manual annotation or dictionary construction for text segmentation, and effectively identifying specialized vocabulary within the knowledge domain of the text.

[0018] The text processing method according to embodiments of this disclosure can be executed by a text processing device, which can be integrated into an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The text processing method can be implemented by a processor calling computer-readable program instructions stored in memory, or it can be executed by a server.

[0019] Figure 1 A flowchart of a text processing method provided in this disclosure embodiment is shown below. Figure 1 The method includes steps S11 to S13.

[0020] Step S11: Obtain the text to be processed and determine the knowledge domain to which the text to be processed belongs.

[0021] The directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is created in advance based on the frequency of each character and the word group formed by each character and its adjacent characters in the domain text corpus to which the text to be processed belongs. The word group formed by each character and its adjacent characters refers to any word formed by any character in the text to be processed and any character adjacent to that character.

[0022] In this embodiment of the disclosure, words have knowledge domain characteristics. Different knowledge domains have different vocabulary systems, such as the military field, the medical field, the public security field, and the financial field. There are significant differences in vocabulary systems between different knowledge domains.

[0023] In step S11, domain text can be obtained as the text to be processed from the pre-collected domain text corpora corresponding to each knowledge domain, or the text to be processed can be obtained by providing a text input interface or other means for the user to input the text to be processed. The user-input text to be processed can carry the knowledge domain attribute to which the text belongs. Further, when obtaining the text to be processed from the domain text corpora of a knowledge domain, the knowledge domain to which the text to be processed belongs can be determined based on the domain text corpus to which the text to be processed belongs. When obtaining the text to be processed through user input, the knowledge domain to which the text to be processed belongs can be determined based on the knowledge domain attribute carried by the text to be processed, such as military, medical, public security, financial, etc. In some embodiments, the knowledge domain to which the text to be processed belongs can also be determined by counting the number of characters appearing in the domain text corpus of any knowledge domain in the text to be processed, and the knowledge domain corresponding to the largest number can be taken as the knowledge domain to which the text to be processed belongs.

[0024] In this embodiment, the number of texts to be processed can be one or more. When there are multiple texts to be processed, the text processing method of this embodiment can be used to process the multiple texts separately or simultaneously to obtain the word segmentation result corresponding to each text. The text to be processed can be any piece of text that needs to be segmented. For example, the text to be processed can be any sentence in a text to be segmented, or it can be obtained by converting user speech during human-computer interaction. This embodiment does not limit the text to be processed.

[0025] Step S12: Obtain the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs.

[0026] In this embodiment of the disclosure, multiple directed acyclic graphs corresponding to different knowledge domains can be created in advance based on domain text corpora of multiple knowledge domains. In step S12, after obtaining the text to be processed and determining its knowledge domain, the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is queried and obtained from the multiple directed acyclic graphs corresponding to different knowledge domains created in advance.

[0027] Among them, the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs includes the directed acyclic graph corresponding to each character in the domain text corpus of the knowledge domain. The directed acyclic graph records the number of times each character in the domain text corpus appears in the domain text corpus, as well as the number of times each character and its adjacent word groups appear in the domain text corpus.

[0028] See Figure 3, which is a schematic diagram of a directed acyclic graph of the knowledge domain to which the text to be processed belongs according to an embodiment of the present application. Assuming that n (n≥1) different characters appear in the domain text corpus of a domain, therefore, for the domain text corpus of this domain, n directed acyclic graphs corresponding to the n characters respectively need to be created, and the n directed acyclic graphs corresponding to the n characters form the directed acyclic graph of the knowledge domain. Taking Figure 3 the directed acyclic graph of the i-th character "mei (meaning beautiful, good in Chinese)" as an example, in the domain text corpus, there are x (x≥1) different characters "zhong (meaning China in Chinese)", "xiu (meaning elegant in Chinese)", ..., "wan (meaning complete in Chinese)" and the like that are located to the left of the character "mei" and adjacent to the character "mei"; , , ..., respectively represent the occurrence times of word groups such as "zhong mei", "xiu mei", ..., "wan mei" in the domain text corpus.

[0029] In the domain text corpus, there are y (y≥1) different characters "hao (meaning good in Chinese)", "li (meaning beautiful in Chinese)", ..., "guo (meaning country in Chinese)" and the like that are located to the right of the character "mei" and adjacent to the character "mei"; , , ..., respectively represent the occurrence times of word groups such as "mei hao", "mei li", ..., "mei guo" in the domain text corpus; dD i represents the occurrence times of the character "mei" in the domain text corpus.

[0030] Based on the counted occurrence times of the character "mei" and the word groups formed by the character "mei" and adjacent characters in the domain text corpus respectively, the directed acyclic graph corresponding to the character "mei" in the domain text corpus as shown in Figure 3 is constructed. Similarly, according to the above construction method of the directed acyclic graph corresponding to the character "mei", the construction of the directed acyclic graphs corresponding to the n different characters respectively in the domain text corpus is completed.

[0031] In one embodiment, the directed acyclic graph of the knowledge domain to which the text to be processed belongs may be executed before executing each step shown in Figure 1 , and of course it may also be executed after step S11 is executed. In the embodiment of the present application, it is assumed to be executed before each step of Figure 1 . The following takes the knowledge domain to which the text to be processed belongs as an example to introduce how to construct a directed acyclic graph corresponding to the knowledge domain. Optionally, creating the directed acyclic graph of the knowledge domain may include steps S101 to S104. As shown in Figure 2 , steps S101 to S104 may be located before steps S11 to S13: Step S101: Obtain the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs. The domain text corpus includes multiple domain texts corresponding to the knowledge domain. The domain text corpus can be a collection of multiple domain texts within the knowledge domain, where each domain text can include one or more characters.

[0032] Step S102: Determine the first occurrence number of each character in the domain text corpus, and the second occurrence number of each word and its adjacent word combination in the domain text corpus.

[0033] After acquiring the domain text corpus, the number of times each character appears in the domain text corpus is counted as the first count, and the number of times each character and its adjacent words appear in the domain text corpus is counted as the second count.

[0034] Step S103: Based on the first occurrence count of each character in the domain text corpus and the second occurrence count of each character and its adjacent word group in the domain text corpus, create a directed acyclic graph corresponding to each character in the domain text corpus.

[0035] Step S104: Based on the directed acyclic graph corresponding to each character in the domain text corpus, obtain the directed acyclic graph corresponding to the knowledge domain.

[0036] When it is necessary to create directed acyclic graphs corresponding to multiple knowledge domains, a directed acyclic graph corresponding to each knowledge domain can be created according to the methods in steps S101 to S104 above.

[0037] Step S13: Based on the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs and the text length of the text to be processed, perform word segmentation on the text to be processed to obtain the word segmentation result corresponding to the text to be processed.

[0038] According to the technical solution of the text processing method provided in this disclosure, by statistically analyzing the frequency of occurrence of each character and any two adjacent character phrases in the domain text corpus, a directed acyclic graph (DAG) is created for each character in the domain text corpus. During text segmentation, the DAG allows for quick querying of the frequency of related character combinations, enabling rapid and accurate text segmentation based on the retrieved frequencies, thus effectively improving the efficiency and accuracy of text segmentation. Furthermore, in the text processing method of this disclosure, both the text and the DAG are divided according to the domain, eliminating the need for manual annotation or dictionary construction for text segmentation, and effectively identifying specialized vocabulary within the knowledge domain of the text.

[0039] In some embodiments, different text lengths may correspond to different word segmentation methods. For to-be-processed texts with different text lengths, different word segmentation methods based on directed acyclic graphs may be adopted to perform word segmentation processing on different to-be-processed texts. For example, for a to-be-processed text with a text length of 0, that is, a text including 0 characters, no word segmentation processing is required, and a word segmentation result representing a null value may be returned at this time; for a to-be-processed text with a text length of 1, that is, a text including 1 character, no word segmentation processing is required, a separator " / " may be added after the 1 character of the to-be-processed text, and a word segmentation result including the 1 character of the to-be-processed text and the subsequent separator " / " may be returned. Illustratively, if the to-be-processed text is "Ni", a word segmentation result represented as "Ni / " is returned; for a to-be-processed text with a text length of 2, that is, a text including 2 characters, word segmentation processing is performed on the to-be-processed text by using a first word segmentation method based on the directed acyclic graph corresponding to the knowledge field to which the to-be-processed text belongs, and a corresponding word segmentation result is obtained; for a to-be-processed text with a text length of 3, that is, a text including 3 characters, word segmentation processing is performed on the to-be-processed text by using a second word segmentation method based on the directed acyclic graph corresponding to the knowledge field to which the to-be-processed text belongs, and a corresponding word segmentation result is obtained; for a to-be-processed text with a text length greater than or equal to 4, that is, a text including 4 or more characters, word segmentation processing is performed on the to-be-processed text by using a third word segmentation method based on the directed acyclic graph corresponding to the knowledge field to which the to-be-processed text belongs, and a corresponding word segmentation result is obtained.

[0040] The following is a detailed introduction to various word segmentation methods in turn: Figure 4 is Figure 1 a flowchart of a specific implementation of step S13 herein. In some embodiments, with reference to Figure 4 , based on the directed acyclic graph corresponding to the knowledge field to which the to-be-processed text belongs and the text length of the to-be-processed text, performing word segmentation processing on the to-be-processed text to obtain the word segmentation result corresponding to the to-be-processed text, that is, step S13, may further include: step S131a to step S131b.

[0041] Step S131a, when the text length of the to-be-processed text is 2, acquiring, from the directed acyclic graph, the occurrence frequency of a phrase formed by two characters in the to-be-processed text, and the occurrence frequency of each of the two characters in the to-be-processed text.

[0042] Illustratively, assuming that the to-be-processed text is "Mei Hao", the occurrence frequency of the character "Mei" in the domain text corpus of the corresponding knowledge field, the occurrence frequency of the character "Hao" in the domain text corpus of the corresponding knowledge field, and the occurrence frequency of the phrase "Mei Hao" in the domain text corpus of the corresponding knowledge field are acquired from the directed acyclic graph corresponding to the knowledge field.

[0043] Step S131b: Based on the number of times each character appears in the text to be processed, and the number of times a word consisting of two characters appears in the text to be processed, perform word segmentation on the text to be processed to obtain the word segmentation result corresponding to the text to be processed.

[0044] Figure 5 for Figure 4 A flowchart of a specific implementation of step S131b is provided in some embodiments. Figure 5 The step of performing word segmentation on the text to be processed based on the number of times each character appears in the text to be processed and the number of times a word consisting of two characters appears in the text to be processed, i.e., step S131b, may further include: steps S131b1 to S131b2.

[0045] Step S131b1: Using a preset algorithm, calculate the probability of two characters forming a word in the text by considering the number of times each character appears in the text and the number of times a word consisting of two characters appears in the text.

[0046] In some embodiments, the preset algorithm includes a logarithmic function. In step S131b1, the frequency of each character in the text to be processed is first multiplied, and the result of the multiplication is compared with the frequency of the word phrase corresponding to the two characters in the text to be processed to obtain a ratio. Then, the logarithmic function is used to perform a logarithmic operation on the ratio to obtain a corresponding logarithmic value, and the calculated logarithmic value is used as the probability of the two characters in the text forming a word. The probability of forming a word represents the likelihood that the two characters in the text to be processed can form a valid word, i.e., a word.

[0047] The logarithmic function is a logarithmic function with base 10, which includes: .in, This indicates the number of times the first character of the two characters in the text to be processed appears in the domain text corpus of the relevant knowledge domain. This indicates the number of times the second character in the two characters of the text to be processed appears in the domain text corpus of the relevant knowledge domain. This represents the number of times each word or phrase consisting of the first and second characters of the text to be processed appears in the domain-specific text corpus of the relevant knowledge domain. The value of represents the probability that the first and second characters of the text to be processed form a valid word. For example, if the first character appears 5 times, the second character appears 10 times, and the phrase containing the first and second characters appears 5 times, then the result of the above multiplication operation is 5. 10=50, the above ratio is 50 / 5=10, then the logarithmic value calculated by using the above logarithmic function is log 10 (10)=1.

[0048] Step S131b2: performing word segmentation processing on the text to be processed based on the word formation probability corresponding to two characters in the text to be processed, so as to obtain a word segmentation result corresponding to the text to be processed.

[0049] When the word formation probability corresponding to two characters in the text to be processed is greater than a preset threshold, it indicates that the probability that the two characters in the text to be processed form a valid word, that is, form a word, is relatively high, then the phrase formed by the two characters in the text to be processed is taken as a word segmentation and added to the word segmentation result corresponding to the text to be processed, so as to obtain the word segmentation result corresponding to the text to be processed; when the word formation probability corresponding to two characters in the text to be processed is less than or equal to the preset threshold, it indicates that the probability that the two characters in the text to be processed form a valid word, that is, form a word, is relatively low, then each of the two characters in the text to be processed is taken as a word segmentation respectively and added to the word segmentation result corresponding to the text to be processed, so as to obtain the word segmentation result corresponding to the text to be processed. In the word segmentation result corresponding to the text to be processed, each word segmentation is sorted sequentially according to the corresponding position in the text to be processed, and a separator " / " is added after each word segmentation, and adjacent word segments are separated by the separator " / ".

[0050] Wherein, the preset threshold can be configured according to actual conditions, which is not particularly limited in the embodiments of the present disclosure. For example, the preset threshold is set to 0.6.

[0051] For example, the text to be processed is "mei hao" (meaning "beautiful" in Chinese), , and respectively represent the number of occurrences of the character "mei", the character "hao", and the phrase "mei hao" in the domain text corpus, the preset threshold is , when , , and satisfy the condition: > , it indicates that the probability that the two characters "mei" and "hao" form a valid word, that is "mei hao", is relatively high, then the phrase formed by the two characters "mei" and "hao" in the text to be processed, that is "mei hao", is taken as a word segmentation, and the separator " / " is added after the word "mei hao", and the word segmentation result represented as "mei hao / " is returned.

[0052] When ≤ indicates that the probability that the two Chinese characters "mei" (beautiful) and "hao" (good) form a valid word, i.e., "meihao" (beautiful), is low. In this case, the word "mei" and the word "hao" in the text to be processed are each segmented as an independent word, and a delimiter " / " is added between "mei" and "hao", and a delimiter " / " is added after "hao", thereby obtaining a segmentation result expressed as "mei / hao / ".

[0053] The foregoing steps S131a to S131b illustrate a first word segmentation method for a to-be-processed text with a text length of 2. For a to-be-processed text with a text length of 2, performing word segmentation processing on the to-be-processed text by using the first word segmentation method based on the occurrence frequency of each character in the to-be-processed text and the occurrence frequency of the word formed by the two characters in the to-be-processed text can effectively improve the efficiency and accuracy of word segmentation processing.

[0054] Figure 6 is Figure 1 a flowchart of a specific embodiment of step S13. In some embodiments, with reference to Figure 6 , based on the directed acyclic graph corresponding to the knowledge domain to which the to-be-processed text belongs and the text length of the to-be-processed text, the step of performing word segmentation processing on the to-be-processed text to obtain the word segmentation result corresponding to the to-be-processed text, that is, step S13, may further include: steps S132a to S132b.

[0055] Step S132a, when the text length of the to-be-processed text is 3, acquiring, from the directed acyclic graph, the respective occurrence frequencies of word groups formed by every two adjacent characters among the three characters of the to-be-processed text.

[0056] For example, assuming that the to-be-processed text is "qu xue xiao" (go to school), the occurrence frequency of the word group formed by the first group of adjacent two characters "qu" (go) and "xue" (study), that is, "qu xue", in the domain text corpus of the corresponding knowledge domain is acquired from the directed acyclic graph corresponding to the knowledge domain; and the occurrence frequency of the word group formed by the second group of adjacent two characters "xue" (study) and "xiao" (school), that is, "xue xiao" (school), in the domain text corpus of the corresponding knowledge domain is acquired.

[0057] Step S132b, performing word segmentation processing on the to-be-processed text according to the respective occurrence frequencies of word groups formed by every two adjacent characters among the three characters of the to-be-processed text, so as to obtain the word segmentation result corresponding to the to-be-processed text.

[0058] In step S132b, when the occurrence frequency of the word group formed by the former group of adjacent two characters among the every two adjacent characters of the three characters in the to-be-processed text is equal to the occurrence frequency of the word group formed by the latter group of adjacent two characters, it indicates that the probability that the three characters of the to-be-processed text form a valid word is relatively high, so the three characters in the to-be-processed text are combined into one word segmentation and added to the word segmentation result corresponding to the to-be-processed text.

[0059] When the number of occurrences of the phrase formed by the current group of two adjacent characters is less than the number of occurrences of the phrase formed by the next group of two adjacent characters, it indicates that the probability that the next group of two adjacent characters forms a valid word is relatively high. Then, the first character in the text to be processed is taken as a word segment, and the two adjacent characters in the next group are combined into a word segment, which is added to the word segmentation result corresponding to the text to be processed.

[0060] When the number of occurrences of the phrase formed by the current group of two adjacent characters is greater than the number of occurrences of the phrase formed by the next group of two adjacent characters, it indicates that the probability that the previous group of two adjacent characters forms a valid word is relatively high. Then, the two adjacent characters in the previous group in the text to be processed are combined into a word segment, and the last character is taken as a word segment, which is added to the word segmentation result corresponding to the text to be processed.

[0061] In the word segmentation result corresponding to the text to be processed, each word segment is sorted sequentially according to their corresponding position order in the text to be processed, and a separator " / " is added after each word segment, and adjacent word segments are separated by the separator " / ".

[0062] By way of example, suppose that the number of occurrences of the phrase "qu xue" formed by the previous group of adjacent characters "qu" and "xue" in the domain text corpus of the corresponding knowledge domain is a, and the number of occurrences of the phrase "xue xiao" formed by the next group of adjacent characters "xue" and "xiao" in the domain text corpus of the corresponding knowledge domain is b. When a = b, "qu xue xiao" is considered as a valid word, "qu xue xiao" is taken as a word segment, the separator " / " is added after the word "qu xue xiao", and the word segmentation result represented as "qu xue xiao / " is returned; when a < b, "xue xiao" is considered as a valid word, "qu" is separately taken as a word segment, "xue xiao" is taken as a word segment, the separator " / " is added after the character "qu" and the word "xue xiao" respectively, and the word segmentation result represented as "qu / xue xiao / " is returned; when a > b, "qu xue" is considered as a valid word, "qu xue" is taken as a word segment, and "xiao" is separately taken as a word segment, the separator " / " is added after the word "qu xue" and the character "xiao" respectively, and the word segmentation result represented as "qu xue / xiao / " is returned.

[0063] The above steps S132a to S132b show the second word segmentation method when the text length of the text to be processed is 3. For the text to be processed with a text length of 3, according to the respective occurrence times of phrases formed by every two adjacent characters in the text to be processed, the second word segmentation method is used to perform word segmentation processing on the text to be processed, which can effectively improve the efficiency and accuracy of word segmentation processing.

[0064] Figure 7 is Figure 1 a flowchart of a specific embodiment of step S13 in , in some embodiments, with reference to Figure 7The step of performing word segmentation on the text to be processed based on the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs and the text length of the text to be processed, i.e., step S13, may further include: steps S133a to S133j.

[0065] Step S133a: When the length of the text to be processed is greater than or equal to 4, start word segmentation traversal.

[0066] Step S133b: Obtain the first to fourth consecutive adjacent characters from the remaining characters to be segmented in the text to be processed, and combine these four characters into a text as the text segment to be traversed.

[0067] For example, suppose the text to be processed is "Xiaoming goes to school at eight o'clock in the morning". The remaining words to be segmented in the text include "Xiaoming goes to school at eight o'clock in the morning". Then the first to fourth consecutive words are "Xiaoming in the morning", which means the text segment being traversed is "Xiaoming in the morning".

[0068] Step S133c: Obtain the number of occurrences of each word group consisting of two adjacent characters in the four characters of the currently traversed text segment from the directed acyclic graph.

[0069] In step S133c, the word groups formed by each pair of adjacent characters in the four characters of the currently traversed text segment include the first word group formed by the first character and the second character, the second word group formed by the second character and the third character, and the third word group formed by the third character and the fourth character. The number of times the first word group appears is 'c', the number of times the second word group appears is 'd', and the number of times the third word group appears is 'e'.

[0070] Taking the text to be processed as "Xiaoming goes to school at eight o'clock in the morning" and the text segment currently being traversed as "Xiaoming in the morning" as an example, in step S133c, the number of occurrences of the first phrase "Xiaoming", the number of occurrences of the second phrase "tomorrow", and the number of occurrences of the third phrase "morning" are obtained from the directed acyclic graph of the knowledge domain.

[0071] Step S133d: Based on the number of occurrences of each word group consisting of two adjacent characters in the four characters of the currently traversed text segment, perform word segmentation on the currently traversed text segment to obtain the word segmentation result corresponding to the currently traversed text segment.

[0072] Step S133e: Update the remaining words to be segmented in the text to be processed based on the word segmentation results corresponding to the currently traversed text segment.

[0073] Specifically, based on the word segmentation results corresponding to the currently traversed text segment, if all four characters in the currently traversed text segment have been segmented and there are no characters whose segmentation is not determined, then the characters in the text to be processed that are in the order following the currently traversed text segment are taken as the remaining characters to be segmented, so as to update the remaining characters to be segmented in the text to be processed.

[0074] If there is an undetermined word segment among the four characters in the currently traversed text segment, then the undetermined word segment in the currently traversed text segment, as well as the character in the text to be processed that is in sequence after the undetermined word segment, are taken as the current remaining words to be segmented, so as to update the current remaining words to be segmented in the text to be processed.

[0075] For example, taking the text to be processed as "Xiaoming goes to school at eight o'clock in the morning", and the text segment currently being traversed as "Xiaoming in the morning", assuming the word segmentation result of the currently traversed text segment is "Xiaoming / morning / ", according to the word segmentation result of the currently traversed text segment, we know that all characters in the currently traversed text segment have been segmented, and the number of characters remaining to be segmented in the currently traversed text segment is 0. Therefore, we know that the remaining characters to be segmented in the text to be processed are "eight o'clock go to school", a total of 7 characters.

[0076] For example, taking the text to be processed as "the morning sun is particularly bright" and the currently traversed text segment as "the morning sun", assuming the word segmentation result of the currently traversed text segment is "the morning sun / the sun", according to the word segmentation result of the currently traversed text segment, it can be seen that the two characters "the sun" in the currently traversed text segment have not been segmented. Therefore, "the sun" and the character "the sun is particularly bright" after "the sun" in the text to be processed are taken as the remaining characters to be segmented in the text to be processed. The remaining characters to be segmented in the text to be processed are "the sun is particularly bright", a total of 7 characters.

[0077] Step S133f: When the number of remaining characters to be segmented in the text to be processed is greater than or equal to 4, return to step S133b until the number of remaining characters to be segmented in the text to be processed is less than 4.

[0078] When the number of remaining characters to be segmented in the text to be processed is greater than or equal to 4, return to step S133b and continue processing the next text segment containing 4 characters; when the number of remaining characters to be segmented in the text to be processed is less than 4, execute step S133g, step S133h, step S133i, or step S133j.

[0079] Step S133g: When the number of remaining characters to be segmented in the text to be processed is 0, determine the segmentation result of the text to be processed based on the segmentation result of the text segment traversed each time.

[0080] In step S133g, the word segmentation results corresponding to each traversed text segment are spliced and combined sequentially in the traversal order to obtain the word segmentation result corresponding to the text to be processed.

[0081] For example, taking the text to be processed "Xiaoming starts studying at eight o'clock" as an example, the word segmentation result of the first traversed text segment is "Xiaoming / eight o'clock / ", and the word segmentation result of the second text segment is "start / studying / ". At this time, the number of remaining characters to be segmented in the text to be processed is 0. The word segmentation result "Xiaoming / eight o'clock / " corresponding to the first text segment and the word segmentation result "start / studying / " corresponding to the second text segment are spliced and combined, and the final word segmentation result corresponding to the text to be processed "Xiaoming starts studying at eight o'clock" is obtained as "Xiaoming / eight o'clock / start / studying / ".

[0082] Step S133h: When the number of remaining characters to be segmented in the text to be processed is 1, the remaining 1 character to be segmented in the text to be processed is regarded as a word segmentation, and added to the word segmentation result corresponding to the text to be processed.

[0083] The word segmentation result corresponding to the text to be processed is determined according to the word segmentation results corresponding to each traversed text segment and the word segmentation corresponding to the current remaining 1 character to be segmented in step S133h. Specifically, the word segmentation results corresponding to each traversed text segment and the word segmentation corresponding to the current remaining 1 character to be segmented are spliced and combined sequentially in the traversal order to obtain the word segmentation result corresponding to the text to be processed.

[0084] For example, taking the text to be processed "Xiaoming gets up at eight o'clock" as an example, the word segmentation result of the first text segment is "Xiaoming / eight o'clock / ". At this time, the remaining character to be segmented in the text to be processed is the character "gets up", so "gets up" is separately regarded as a word segmentation "gets up / ", and the word segmentation result "Xiaoming / eight o'clock / " corresponding to the first text segment and the word segmentation result "gets up / " of the last character are spliced and combined in order, and the word segmentation result corresponding to the text to be processed is obtained as "Xiaoming / eight o'clock / gets up / ".

[0085] Step S133i: When the number of remaining characters to be segmented in the text to be processed is 2, the first word segmentation method is used to perform word segmentation processing on the two remaining characters to be segmented to obtain a first word segmentation result, and the word segmentation result corresponding to the text to be processed is determined according to the first word segmentation result and the word segmentation results corresponding to each traversed text segment.

[0086] Wherein, the first word segmentation method includes the foregoing step S131a to step S131b. Determining the word segmentation result corresponding to the text to be processed according to the first word segmentation result and the word segmentation result corresponding to each traversed text segment comprises: performing splicing and combination in sequence according to the traversal order on the word segmentation results corresponding to each traversed text segment and the first word segmentation result corresponding to the current remaining two characters to be segmented, to obtain the word segmentation result corresponding to the text to be processed.

[0087] For example, taking "Xiaoming gets up at eight o'clock" as the text to be processed, the word segmentation result of the first text segment is "Xiaoming / eight o'clock / ", and at this time, the remaining characters to be segmented in the text to be processed are two characters "gets up". For the two-character text, word segmentation is performed by using the foregoing first word segmentation method, and the corresponding word segmentation result obtained is "gets up / ". The word segmentation result "Xiaoming / eight o'clock / " corresponding to the first text segment and the word segmentation result "gets up / " of the last two characters are spliced and combined in order, and the word segmentation result corresponding to the text to be processed obtained is "Xiaoming / eight o'clock / gets up / ". Wherein, for the description of the first word segmentation method, reference may be made to the relevant description in the foregoing steps S131a to S131b and steps S131b1 to S131b2, which will not be repeated herein.

[0088] Step S133j: when the number of currently remaining characters to be segmented in the text to be processed is 3, performing word segmentation processing on the currently remaining three characters to be segmented by using a second word segmentation method to obtain a second word segmentation result, and determining the word segmentation result corresponding to the text to be processed according to the second word segmentation result and the word segmentation result corresponding to each traversed text segment.

[0089] Wherein, the second word segmentation method includes the foregoing step S132a to step S132b. Determining the word segmentation result corresponding to the text to be processed according to the second word segmentation result and the word segmentation result corresponding to each traversed text segment comprises: performing splicing and combination in sequence according to the traversal order on the word segmentation results corresponding to each traversed text segment and the second word segmentation result corresponding to the current remaining three characters to be segmented, to obtain the word segmentation result corresponding to the text to be processed.

[0090] For example, taking "Xiaoming goes to school at eight o'clock" as the text to be processed, the word segmentation result of the first text segment is "Xiaoming / eight o'clock / ", and at this time, the remaining characters to be segmented in the text to be processed are three characters "goes to school". For the three-character text, word segmentation is performed by using the foregoing second word segmentation method, and the corresponding word segmentation result obtained is "goes / to school / ". The word segmentation result "Xiaoming / eight o'clock / " corresponding to the first text segment and the word segmentation result "goes / to school / " of the last three characters are spliced and combined in order, and the word segmentation result corresponding to the text to be processed obtained is "Xiaoming / eight o'clock / goes / to school / ". Wherein, for the description of the second word segmentation method, reference may be made to the relevant description in the foregoing steps S132a to S132b, which will not be repeated herein.

[0091] Figure 8 is Figure 7 a flow chart of a specific implementation of step S133d in. With reference to Figure 8 , in some embodiments, among the four characters of the currently traversed text segment, the word groups corresponding to every two adjacent characters comprise a first word group formed by the 1st character and the 2nd character, a second word group formed by the 2nd character and the 3rd character, and a third word group formed by the 3rd character and the 4th character. Let the number of occurrences corresponding to the first word group be c, the number of occurrences corresponding to the second word group be d, and the number of occurrences corresponding to the third word group be e. The step of performing word segmentation on the currently traversed text segment according to the number of occurrences of the word groups formed by every two adjacent characters among the four characters of the currently traversed text segment, that is step S133d, may further comprise: Step S133d1: When c=d=e, form the four characters of the currently traversed text segment into a word segmentation, and add it to the word segmentation result corresponding to the currently traversed text segment.

[0092] When c=d=e, it indicates that the probability that the four characters of the currently traversed text segment form a word is relatively high, so the four characters of the currently traversed text segment are formed into a word segmentation, and a separator " / " is added after the last character of the currently traversed text segment to obtain the word segmentation result corresponding to the current text segment.

[0093] For example, assuming the text to be processed is "ABCDEFG", the currently traversed text segment is "ABCD", the number of occurrences c of the first word group "AB" is equal to the number of occurrences d of the second word group "BC", and is also equal to the number of occurrences e of the third word group "CD", which indicates that the probability that "ABCD" forms a valid word is relatively high. Therefore, "ABCD" is formed into a word as a word segmentation, and the separator " / " is added after the character "D" to obtain the word segmentation result "ABCD / " corresponding to the currently traversed text segment. At this time, there is no character with undetermined word segmentation in the currently traversed text segment, and the current remaining characters to be segmented in the text to be processed are updated to "EFG".

[0094] Step S133d2: When c<d≤e, take the 1st character in the currently traversed text segment as a word segmentation, add it to the word segmentation result corresponding to the currently traversed text segment, and take the remaining 3 characters as the current remaining characters to be segmented in the text to be processed.

[0095] When c<d≤e, it indicates that the probability that the 1st character and the 2nd character of the currently traversed text segment form a word is relatively low, and the 2nd to 4th characters may form a word, but it still needs to be judged in combination with the following text. Therefore, the 1st character in the currently traversed text segment is taken as a word segmentation, and a delimiter " / " is added after the 1st character of the currently traversed text segment to obtain the word segmentation result corresponding to the currently traversed text segment. The remaining 3 characters are taken as the undetermined segmented characters in the currently traversed text segment, and the 3 undetermined segmented characters in the currently traversed text segment and the characters after the 3 characters in the text to be processed are taken as the current remaining characters to be segmented in the text to be processed, so as to update the current remaining characters to be segmented in the text to be processed.

[0096] For example, assuming the text to be processed is "ABCDEFG", the currently traversed text segment is "ABCD", the occurrence frequency c of the first phrase "AB" is less than the occurrence frequency d of the second phrase "BC", and the occurrence frequency d of the second phrase "BC" is less than or equal to the occurrence frequency e of the third phrase "CD", which indicates that the probability that "A" and "B" form a valid word is relatively low, while "B", "C" and "D" may form a word, but it still needs to be judged in combination with the following text. Therefore, the 1st character "A" in the currently traversed text segment is taken as a word segmentation, and a delimiter " / " is added after "A" to obtain the word segmentation result "A / " corresponding to the currently traversed text segment. At this time, there are 3 undetermined segmented characters "BCD" in the currently traversed text segment, and the current characters to be segmented in the text to be processed are updated to "BCDEFG".

[0097] Step S133d3: When c=d>e, the 1st to 3rd characters in the currently traversed text segment are taken as a word segmentation, added to the word segmentation result corresponding to the currently traversed text segment, and the remaining 1 character is taken as the current remaining character to be segmented in the text to be processed.

[0098] When c=d>e, it indicates that the probability that the 1st to 3rd characters of the currently traversed text segment form a word is relatively high. Therefore, the 1st to 3rd characters in the currently traversed text segment are combined into a valid word as a word segmentation, and a delimiter " / " is added after the 3rd character of the currently traversed text segment to obtain the word segmentation result corresponding to the currently traversed text segment. The remaining 1 character is taken as the undetermined segmented character in the currently traversed text segment, and the 1 undetermined segmented character in the currently traversed text segment and the character after this character in the text to be processed are taken as the current remaining character to be segmented in the text to be processed, so as to update the current remaining character to be segmented in the text to be processed.

[0099] For example, assuming the text to be processed is "ABCDEFG", the currently traversed text segment is "ABCD", the occurrence number c of the first phrase "AB" is equal to the occurrence number d of the second phrase "BC", and the occurrence number d of the second phrase "BC" is greater than the occurrence number e of the third phrase "CD", which indicates that the probability that "A", "B" and "C" form a valid word is relatively high. Therefore, "ABC" is taken as a word segmentation, and a separator " / " is added after "ABC", so as to obtain the word segmentation result "ABC / " corresponding to the currently traversed text segment. At this time, there is 1 undetermined word segmentation character "D" in the currently traversed text segment, and the current character to be segmented in the text to be processed is updated to "DEFG".

[0100] Step S133d4: when c=d<e or c>d, taking the 1st character to the 2nd character in the current text segment as a word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 2 characters as the current remaining characters to be segmented in the text to be processed.

[0101] When c=d<e or c>d, it indicates that the probability that the 1st character to the 2nd character of the currently traversed text segment form a word is relatively high. Therefore, the 1st character to the 2nd character in the currently traversed text segment are combined into a valid word as a word segmentation, a separator " / " is added after the word segmentation to obtain the word segmentation result corresponding to the currently traversed text segment, the remaining 2 characters are taken as the undetermined word segmentation characters in the currently traversed text segment, and the 2 undetermined word segmentation characters in the currently traversed text segment and the characters after the 2 characters in the text to be processed are taken as the current remaining characters to be segmented in the text to be processed, so as to update the current remaining characters to be segmented in the text to be processed.

[0102] For example, assuming the text to be processed is "ABCDEFG", the currently traversed text segment is "ABCD", the occurrence number c of the first phrase "AB" is greater than the occurrence number d of the second phrase "BC", or the occurrence number c of the first phrase "AB" is equal to the occurrence number d of the second phrase "BC", and the occurrence number d of the second phrase "BC" is less than the occurrence number e of the third phrase "CD", which indicates that the probability that "A" and "B" form a valid word is relatively high, while the word segmentation of "CD" cannot be determined and needs to be judged in combination with the following text. Therefore, "AB" is taken as a word segmentation, and a separator " / " is added after "AB", so as to obtain the word segmentation result "AB / " corresponding to the currently traversed text segment. At this time, there are 2 undetermined word segmentation characters "CD" in the currently traversed text segment, and the current character to be segmented in the text to be processed is updated to "CDEFG".

[0103] Step S133d5: When d>c or d>e, take the first character in the currently traversed text segment as a word segment, take the second and third characters in the currently traversed text segment as a word segment, add them to the word segmentation result corresponding to the currently traversed text segment, and take the remaining character as the remaining word to be segmented in the text to be processed.

[0104] When d>c or d>e, it means that the probability of the first and second characters of the currently traversed text segment forming a word is relatively low, while the probability of the second and third characters forming a word is relatively high. Therefore, the first character of the currently traversed text segment is taken as a word segment, and the second and third characters of the currently traversed text segment form a valid word segment as a word segment. A separator " / " is added after the first character, and a separator " / " is added after the third character to obtain the word segmentation result corresponding to the currently traversed text segment. The remaining character is taken as the character whose word segmentation is not determined in the currently traversed text segment. The character whose word segmentation is not determined in the currently traversed text segment, and the characters after the character in the text to be processed are taken as the remaining characters to be segmented in the text to be processed, so as to update the remaining characters to be segmented in the text to be processed.

[0105] For example, suppose the text to be processed is “ABCDEFG”, and the currently traversed text segment is “ABCD”. The number of times the first word group “AB” appears, c, is less than the number of times the second word group “BC” appears, d, or the number of times the second word group “BC” appears, d, is greater than the number of times the third word group “CD” appears, e. This means that the probability of “A” and “B” forming a valid word is relatively small, while the probability of “B” and “C” forming a word is relatively large. Therefore, “A” is treated as a word segment, and a separator “ / ” is added after “A”. “BC” is treated as a word segment, and a separator “ / ” is added after “BC” to obtain the word segmentation result “A / BC / ” corresponding to the currently traversed text segment. At this time, there is one character “D” in the currently traversed text segment whose word segmentation is not determined. The character to be segmented in the text to be processed is updated to “DEFG”.

[0106] Steps S133a to S133j above illustrate the third word segmentation method when the text length of the text to be processed is greater than or equal to 4. For texts with a length greater than or equal to 4, using the above third word segmentation method to segment the text can effectively improve the efficiency and accuracy of word segmentation.

[0107] To optimize and iterate the constructed directed acyclic graph (DAG) and further improve the accuracy of text segmentation using DAG, in some embodiments, after obtaining the segmentation result of the text based on the DAG corresponding to the knowledge domain to which the text belongs and the text length of the text to be processed (i.e., after step S13), the text processing method may further include: obtaining the target words in the text to be processed that have segmentation errors; adjusting the frequency of each word group formed by two adjacent characters in the DAG corresponding to the knowledge domain to which the text belongs, so as to update the DAG corresponding to the knowledge domain to which the text belongs; and re-segmenting the text to be processed based on the updated DAG corresponding to the knowledge domain to which the text belongs and the text length of the text to be processed.

[0108] In some embodiments, for the word segmentation results of the text to be processed, the target words in the word segmentation results that contain segmentation errors can be identified through manual error correction.

[0109] In some embodiments of this disclosure, the text to be processed and the domain text are both Chinese text.

[0110] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0111] In addition, this disclosure also provides a text processing apparatus, an electronic device, and a computer-readable storage medium. The above-mentioned text processing apparatus, electronic device, and computer-readable storage medium can all be used to implement any of the text processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0112] Figure 9 This is a block diagram of a text processing apparatus provided in an embodiment of the present disclosure.

[0113] Reference Figure 9 This disclosure provides a text processing device 200, which includes an acquisition unit 201 and a processing unit 202.

[0114] The acquisition unit 201 is used to: acquire the text to be processed and determine the knowledge domain to which the text to be processed belongs; and acquire the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs.

[0115] The processing unit 202 is used to perform word segmentation on the text to be processed based on the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs and the text length of the text to be processed, so as to obtain the word segmentation result corresponding to the text to be processed.

[0116] The directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is created in advance based on the number of times each character and each word group composed of each character and its adjacent characters appear in the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs.

[0117] In one embodiment, the processing unit 202 is further configured to: Obtain the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs. The domain text corpus includes multiple domain texts of the knowledge domain, and each domain text includes multiple characters. Determine the first occurrence number of each character in the plurality of domain texts in the domain text corpus, and the second occurrence number of each word in the plurality of domain texts and the word group formed by each character and its adjacent characters in the domain text corpus; Create a directed acyclic graph for each character based on the first and second counts; Based on the directed acyclic graph corresponding to each character, the directed acyclic graph corresponding to the knowledge domain is obtained.

[0118] In one embodiment, when the processing unit 202 performs word segmentation on the text to be processed based on the directed acyclic graph and the text length of the text to be processed, and obtains the word segmentation result corresponding to the text to be processed, it performs the following steps: When the length of the text to be processed is 2, the number of occurrences of the two-character phrases in the text to be processed and the number of occurrences of each character in the two-character phrases in the text to be processed are obtained from the directed acyclic graph. Based on the number of times each character appears in the text to be processed, and the number of times two-character phrases appear in the text to be processed, the text to be processed is segmented into words to obtain the segmentation results corresponding to the text to be processed.

[0119] In one embodiment, when the processing unit 202 performs word segmentation on the text to be processed based on the number of times each character appears in the text to be processed and the number of times a word consisting of two characters appears in the text to be processed to obtain the word segmentation result corresponding to the text to be processed, it performs the following steps: The algorithm is used to calculate the probability of two characters forming a word in the text by calculating the number of times each character appears in the text and the number of times a word consisting of two characters appears in the text. The text to be processed is segmented based on the word formation probability to obtain the segmentation result corresponding to the text to be processed.

[0120] In one embodiment, when the processing unit 202 performs word segmentation on the text to be processed based on the word formation probability to obtain the word segmentation result corresponding to the text to be processed, it performs the following steps: When the probability of forming a word is greater than a preset threshold, the word group consisting of two characters in the text to be processed is taken as a word segmentation and added to the word segmentation result; When the probability of forming a word is less than or equal to a preset threshold, each of the two characters in the text to be processed is treated as a separate word segment and added to the word segmentation result.

[0121] In one embodiment, the preset algorithm includes a logarithmic function. When the processing unit 202 calculates the probability of two characters forming a word in the text to be processed based on the number of times each character appears in the text to be processed and the number of times a word consisting of two characters appears in the text to be processed using the preset algorithm, the following steps are performed: The number of occurrences of each character in the text to be processed is multiplied together, and the result of the multiplication is compared with the number of occurrences of word groups consisting of two characters in the text to be processed to obtain the ratio. The logarithmic function is used to perform a logarithmic operation on the ratio to obtain a logarithmic value, and the calculated logarithmic value is used as the probability of two characters forming words in the text to be processed.

[0122] In one embodiment, when the processing unit 202 performs word segmentation on the text to be processed based on the directed acyclic graph and the text length of the text to be processed, and obtains the word segmentation result corresponding to the text to be processed, it performs the following steps: When the length of the text to be processed is 3, obtain the number of times each word group consisting of every two adjacent characters in the three characters of the text to be processed appears from the directed acyclic graph; The text to be processed is segmented into words based on the frequency of occurrence of each word group consisting of two adjacent characters in the three characters of the text to be processed, so as to obtain the segmentation result corresponding to the text to be processed.

[0123] In one embodiment, when the processing unit 202 performs word segmentation on the text to be processed based on the number of occurrences of each word group consisting of two adjacent characters in the three characters of the text to be processed, in order to obtain the word segmentation result corresponding to the text to be processed, it performs the following steps: When the number of occurrences of the word group consisting of the first pair of adjacent characters in each of the three characters in the text to be processed is equal to the number of occurrences of the word group consisting of the second pair of adjacent characters, the three characters in the text to be processed are combined into a word segment and added to the word segmentation result. If the number of occurrences of a word group consisting of two adjacent characters in the current group is less than the number of occurrences of a word group consisting of two adjacent characters in the next group, then the first character in the text to be processed is taken as a word segment, and the two adjacent characters in the next group are combined into a word segment and added to the word segmentation result. If the number of occurrences of a word group consisting of two adjacent characters in the current group is greater than the number of occurrences of a word group consisting of two adjacent characters in the next group, then the two adjacent characters in the previous group in the text to be processed will be combined into a word segment, and the last character will be added as a word segment to the word segmentation result.

[0124] In one embodiment, when the processing unit 202 performs word segmentation on the text to be processed based on the directed acyclic graph and the text length of the text to be processed, and obtains the word segmentation result corresponding to the text to be processed, it performs the following steps: When the length of the text to be processed is greater than or equal to 4, the word segmentation traversal begins; From the remaining words to be segmented in the text to be processed, obtain four consecutive adjacent words, and use the text composed of these four words as the text segment to be traversed. Obtain the number of occurrences of each word group consisting of two adjacent characters in the currently traversed text segment from the directed acyclic graph; Based on the frequency of occurrence of word groups consisting of every two adjacent characters in the currently traversed text segment, the text segment is segmented into words to obtain the segmentation result corresponding to the currently traversed text segment. Based on the word segmentation results corresponding to the currently traversed text segment, update the remaining words to be segmented in the text to be processed; When the number of remaining characters to be segmented in the text to be processed is greater than or equal to 4, return to the step of obtaining four consecutive adjacent characters from the remaining characters to be segmented in the text to be processed and combining these four characters into a text as the current text segment, until the number of remaining characters to be segmented in the text to be processed is less than 4.

[0125] In one embodiment, the processing unit 202 is further configured to: When the number of remaining characters to be segmented in the text to be processed is 0, the segmentation result corresponding to the text to be processed is determined according to the segmentation result corresponding to the text segment in each iteration; when the number of currently remaining characters to be segmented in the text to be processed is 1, taking the currently remaining characters to be segmented in the text to be processed as a word segmentation, and adding the word segmentation to the word segmentation result corresponding to the text to be processed; when the number of currently remaining characters to be segmented in the text to be processed is 2, performing word segmentation processing on the two characters of the currently remaining characters to be segmented according to the occurrence times of each character in the currently remaining characters to be segmented and the occurrence times of the phrase formed by the two characters in the currently remaining characters to be segmented, so as to obtain a first word segmentation result corresponding to the two characters of the currently remaining characters to be segmented; and determining the word segmentation result corresponding to the text to be processed according to the first word segmentation result and the word segmentation result corresponding to each traversed text segment; when the number of all currently remaining characters to be segmented in the text to be processed is 3, performing word segmentation processing on the three characters of the currently remaining characters to be segmented according to the respective occurrence times of phrases formed by every two adjacent characters in the three characters of the currently remaining characters to be segmented, so as to obtain a second word segmentation result corresponding to the three characters of the currently remaining characters to be segmented; and determining the word segmentation result corresponding to the text to be processed according to the second word segmentation result and the word segmentation result corresponding to each traversed text segment.

[0126] in one embodiment, among the four characters of the currently traversed text segment, the phrases corresponding to every two adjacent characters comprise a first phrase formed by a first character and a second character, a second phrase formed by the second character and a third character, and a third phrase formed by the third character and a fourth character, wherein the occurrence number corresponding to the first phrase is c, the occurrence number corresponding to the second phrase is d, and the occurrence number corresponding to the third phrase is e; when the processing unit 202 performs word segmentation processing on the current text segment according to the respective occurrence times of phrases formed by every two adjacent characters in the four characters of the currently traversed text segment, the following steps are executed: when c=d=e, forming the four characters of the currently traversed text segment into a word segmentation, and adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment; when c<d≤e, taking the first character in the currently traversed text segment as a word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 3 characters as the currently remaining characters to be segmented in the text to be processed; when c=d>e, taking the first character to the third character in the currently traversed text segment as a word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 1 character as the currently remaining character to be segmented in the text to be processed; when c=d<e or c>d, taking the first character to the second character in the currently traversed text segment as a word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 2 characters as the currently remaining characters to be segmented in the text to be processed; When d>c or d>e, the first character in the currently traversed text segment is taken as a word segment, the second and third characters in the currently traversed text segment are taken as a word segment, and added to the word segmentation result corresponding to the currently traversed text segment. The remaining character is taken as the remaining character to be segmented in the text to be processed.

[0127] In one embodiment, the acquisition unit 201 is further configured to acquire target words in the text to be processed that have segmentation errors; The processing unit 202 is further configured to adjust the frequency of each word group formed by two adjacent characters in the target word in the directed acyclic graph, so as to update the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs; and re-segment the text to be processed based on the updated directed acyclic graph and the text length of the text to be processed.

[0128] The text processing device 200 provided in this embodiment has the advantages of simple structure, easy operation, fast text processing speed, and simple optimization and iteration of directed acyclic graphs, which can effectively save manpower and material resources.

[0129] The text processing apparatus 200 provided in this embodiment is used to implement the text processing method provided in any of the above embodiments. For specific related descriptions, please refer to the descriptions in the text processing methods of any of the above embodiments, which will not be repeated here.

[0130] Figure 10 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0131] Reference Figure 10 This disclosure provides an electronic device, which includes: at least one processor 301; at least one memory 302; and one or more I / O interfaces 303 connected between the processor 301 and the memory 302; wherein the memory 302 stores one or more computer programs that can be executed by the at least one processor 301, and the one or more computer programs are executed by the at least one processor 301 to enable the at least one processor 301 to perform the above-described text processing method.

[0132] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the above-described text processing method. The computer-readable storage medium may be volatile or non-volatile.

[0133] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described text processing method.

[0134] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0135] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0136] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0137] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0138] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0139] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0140] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0141] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0143] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A text processing method, characterized in that, include: Obtain the text to be processed and determine the knowledge domain to which the text to be processed belongs; Obtain the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs; Based on the text length of the text to be processed, the frequency of word occurrences in the text is queried from the directed acyclic graph. According to the frequency and a preset rule corresponding to the text length, the text is segmented to obtain the word segmentation result. Specifically, in step 1, when the text length of the text to be processed is 2, the frequency of word occurrences includes the frequency of two-character phrases and the frequency of each individual character within those two characters. In step 2, when the text length of the text to be processed is 3, the frequency of word occurrences includes the frequency of each adjacent two-character phrase within the three-character phrases. The number of occurrences; Step 3: When the length of the text to be processed is greater than or equal to 4, the first four consecutive adjacent characters are obtained from the remaining text to be segmented in a traversal manner as the current text segment. The number of occurrences of the characters includes the number of occurrences of each word group composed of two adjacent characters in the current text segment. Based on the segmentation result of the current text segment, the remaining characters to be segmented in the text to be processed are updated until the number of remaining characters to be segmented is less than 4. When the number of remaining characters to be segmented is 0 or 1, no segmentation is performed. When the number of remaining characters to be segmented is 2, Step 1 is executed. When the number of remaining characters to be segmented is 3, Step 2 is executed. The directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is created in advance based on the number of times each character and each word group composed of each character and its adjacent characters appear in the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs.

2. The method according to claim 1, characterized in that, Before acquiring the text to be processed, the method further includes: Obtain the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs. The domain text corpus includes multiple domain texts of the knowledge domain, and each domain text includes multiple characters. Determine the first occurrence number of each character in the plurality of domain texts in the domain text corpus, and the second occurrence number of each word in the plurality of domain texts and the word group formed by each character and its adjacent characters in the domain text corpus; Create a directed acyclic graph for each character based on the first and second counts; Based on the directed acyclic graph corresponding to each character, the directed acyclic graph corresponding to the knowledge domain is obtained.

3. The method according to claim 1, characterized in that, The step involves querying the frequency of words in the text to be processed from the directed acyclic graph based on the text length of the text to be processed, and then segmenting the text according to the frequency and a preset rule corresponding to the text length to obtain the word segmentation result, including: When the length of the text to be processed is 2, the number of occurrences of the two-character phrases in the text to be processed and the number of occurrences of each character in the two-character phrases in the text to be processed are obtained from the directed acyclic graph. Based on the number of times each character appears in the text to be processed, and the number of times two-character phrases appear in the text to be processed, the text to be processed is segmented into words to obtain the segmentation results corresponding to the text to be processed.

4. The method according to claim 3, characterized in that, The step of performing word segmentation on the text to be processed based on the frequency of each character in the text to be processed and the frequency of word groups consisting of two characters in the text to be processed to obtain the word segmentation result corresponding to the text to be processed includes: The algorithm is used to calculate the probability of two characters forming a word in the text by calculating the number of times each character appears in the text and the number of times a word consisting of two characters appears in the text. The text to be processed is segmented based on the word formation probability to obtain the segmentation result corresponding to the text to be processed.

5. The method according to claim 4, characterized in that, The step of segmenting the text to be processed based on the word formation probability to obtain the segmentation result corresponding to the text to be processed includes: When the probability of forming a word is greater than a preset threshold, the word group consisting of two characters in the text to be processed is taken as a word segmentation and added to the word segmentation result; When the probability of forming a word is less than or equal to a preset threshold, each of the two characters in the text to be processed is treated as a separate word segment and added to the word segmentation result.

6. The method according to claim 5, characterized in that, The preset algorithm includes a logarithmic function. The algorithm calculates the probability of two characters forming a word in the text by performing calculations based on the frequency of each character in the text and the frequency of word pairs in the text. This includes: The number of occurrences of each character in the text to be processed is multiplied together, and the result of the multiplication is compared with the number of occurrences of word groups consisting of two characters in the text to be processed to obtain the ratio. The logarithmic function is used to perform a logarithmic operation on the ratio to obtain a logarithmic value, and the calculated logarithmic value is used as the probability of two characters forming words in the text to be processed.

7. The method according to claim 1, characterized in that, The step involves querying the frequency of words in the text to be processed from the directed acyclic graph based on the text length of the text to be processed, and then segmenting the text according to the frequency and a preset rule corresponding to the text length to obtain the word segmentation result, including: When the length of the text to be processed is 3, obtain the number of times each word group consisting of every two adjacent characters in the three characters of the text to be processed appears from the directed acyclic graph; The text to be processed is segmented into words based on the frequency of occurrence of each word group consisting of two adjacent characters in the three characters of the text to be processed, so as to obtain the segmentation result corresponding to the text to be processed.

8. The method according to claim 7, characterized in that, The step of segmenting the text into words based on the frequency of occurrence of each word group consisting of two adjacent characters in the three characters of the text to be processed, in order to obtain the segmentation result corresponding to the text to be processed, includes: When the number of occurrences of the word group consisting of the first pair of adjacent characters in each of the three characters in the text to be processed is equal to the number of occurrences of the word group consisting of the second pair of adjacent characters, the three characters in the text to be processed are combined into a word segment and added to the word segmentation result. If the number of occurrences of a word group consisting of two adjacent characters in the current group is less than the number of occurrences of a word group consisting of two adjacent characters in the next group, then the first character in the text to be processed is taken as a word segment, and the two adjacent characters in the next group are combined into a word segment and added to the word segmentation result. If the number of occurrences of a word group consisting of two adjacent characters in the current group is greater than the number of occurrences of a word group consisting of two adjacent characters in the next group, then the two adjacent characters in the previous group in the text to be processed will be combined into a word segment, and the last character will be added as a word segment to the word segmentation result.

9. The method according to claim 1, characterized in that, The step involves querying the frequency of words in the text to be processed from the directed acyclic graph based on the text length of the text to be processed, and then segmenting the text according to the frequency and a preset rule corresponding to the text length to obtain the word segmentation result, including: When the length of the text to be processed is greater than or equal to 4, the word segmentation traversal begins; From the remaining words to be segmented in the text to be processed, obtain four consecutive adjacent words, and use the text composed of these four words as the text segment to be traversed. Obtain the number of occurrences of each word group consisting of every two adjacent characters in the currently traversed text segment from the directed acyclic graph; Based on the frequency of occurrence of word groups consisting of every two adjacent characters in the currently traversed text segment, the text segment is segmented into words to obtain the segmentation result corresponding to the currently traversed text segment. Based on the word segmentation results corresponding to the currently traversed text segment, update the remaining words to be segmented in the text to be processed; When the number of remaining characters to be segmented in the text to be processed is greater than or equal to 4, return to the step of obtaining four consecutive adjacent characters from the remaining characters to be segmented in the text to be processed and combining these four characters into a text as the current text segment, until the number of remaining characters to be segmented in the text to be processed is less than 4.

10. The method according to claim 9, characterized in that, After updating the remaining characters to be segmented in the text to be processed based on the word segmentation result corresponding to the currently traversed text segment, the method further includes: When the number of characters remaining to be segmented in the text to be processed is 0, the segmentation result corresponding to the text to be processed is determined according to the segmentation result corresponding to the text segment in each iteration; When the number of remaining characters to be segmented in the text to be processed is 1, the remaining characters to be segmented in the text to be processed are treated as a single segmentation and added to the segmentation result corresponding to the text to be processed. When the number of currently remaining characters to be segmented in the text to be processed is 2, performing word segmentation processing on the two characters of the currently remaining characters to be segmented according to the occurrence frequency of each character in the currently remaining characters to be segmented and the occurrence frequency of the phrase formed by the two characters in the currently remaining characters to be segmented, to obtain a first word segmentation result corresponding to the two characters of the currently remaining characters to be segmented; and determining the word segmentation result corresponding to the text to be processed according to the first word segmentation result and the word segmentation result corresponding to each traversed text segment; When the number of all currently remaining characters to be segmented in the text to be processed is 3, performing word segmentation processing on the three characters of the currently remaining characters to be segmented according to the respective occurrence frequencies of phrases formed by every two adjacent characters among the three characters of the currently remaining characters to be segmented, to obtain a second word segmentation result corresponding to the three characters of the currently remaining characters to be segmented; and determining the word segmentation result corresponding to the text to be processed according to the second word segmentation result and the word segmentation result corresponding to each traversed text segment.

11. The method according to claim 10, characterized in that, The phrases corresponding to every two adjacent characters among the four characters of the currently traversed text segment include a first phrase formed by the 1st character and the 2nd character, a second phrase formed by the 2nd character and the 3rd character, and a third phrase formed by the 3rd character and the 4th character, wherein the occurrence frequency corresponding to the first phrase is c, the occurrence frequency corresponding to the second phrase is d, and the occurrence frequency corresponding to the third phrase is e; The performing word segmentation processing on the current text segment according to the respective occurrence frequencies of phrases formed by every two adjacent characters among the four characters of the currently traversed text segment includes: when c=d=e, combining the four characters of the currently traversed text segment into one word segmentation, and adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment; when c<d≤e, taking the 1st character in the currently traversed text segment as one word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 3 characters as the currently remaining characters to be segmented in the text to be processed; when c=d>e, taking the 1st character to the 3rd character in the currently traversed text segment as one word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 1 character as the currently remaining character to be segmented in the text to be processed; when c=d<e or c>d, taking the 1st character to the 2nd character in the currently traversed text segment as one word segmentation, adding the word segmentation to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 2 characters as the currently remaining characters to be segmented in the text to be processed; when d>c or d>e, taking the 1st character in the currently traversed text segment as one word segmentation, taking the 2nd character to the 3rd character in the currently traversed text segment as one word segmentation, adding the two word segmentations to the word segmentation result corresponding to the currently traversed text segment, and taking the remaining 1 character as the currently remaining character to be segmented in the text to be processed.

12. The method according to any one of claims 1-11, characterized in that, after obtaining the word segmentation result corresponding to the text to be processed, the method further includes: acquiring a target word with word segmentation error in the text to be processed; In the directed acyclic graph, the frequency of each word group formed by two adjacent characters in the target word is adjusted to update the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs; Based on the updated directed acyclic graph and the text length of the text to be processed, the text to be processed is re-segmented.

13. A text processing device, characterized in that, include: An acquisition unit is used to acquire the text to be processed and determine the knowledge domain to which the text to be processed belongs; The acquisition unit is also used to acquire the directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs; The processing unit is configured to query the frequency of word occurrences in the directed acyclic graph based on the text length of the text to be processed, and segment the text according to the frequency and a preset rule corresponding to the text length to obtain the word segmentation result. Specifically, in step 1, when the text length of the text to be processed is 2, the frequency of word occurrences includes the frequency of occurrences of word pairs consisting of two characters and the frequency of occurrences of each individual character within a two-character phrase. In step 2, when the text length of the text to be processed is 3, the frequency of word occurrences includes the frequency of occurrences of words consisting of every two adjacent characters within a three-character phrase. The number of times each word phrase appears; Step 3: When the length of the text to be processed is greater than or equal to 4, the first four consecutive adjacent characters are obtained from the remaining text to be segmented in a traversal manner as the current text segment. The number of times each word appears includes the number of times each word phrase formed by two adjacent characters appears in the current text segment. Based on the segmentation result of the current text segment, the remaining characters to be segmented in the text to be processed are updated until the number of remaining characters to be segmented is less than 4. When the number of remaining characters to be segmented is 0 or 1, no segmentation is performed. When the number of remaining characters to be segmented is 2, Step 1 is executed. When the number of remaining characters to be segmented is 3, Step 2 is executed. The directed acyclic graph corresponding to the knowledge domain to which the text to be processed belongs is created in advance based on the number of times each character and each word group composed of each character and its adjacent characters appear in the domain text corpus corresponding to the knowledge domain to which the text to be processed belongs.

14. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the text processing method as described in any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text processing method as described in any one of claims 1-12.

Images