A text segmentation method, system, computer device, and storage medium

Abstract

This invention relates to the field of artificial intelligence technology, providing a text segmentation method, system, computer device, and storage medium, comprising: acquiring the text to be segmented; dividing each sentence of the text into multiple semantic blocks according to a left-to-right order, with the end point as the boundary; if two consecutive characters do not have a connected record in the vocabulary of the text to be segmented, then the preceding character of the two consecutive characters is recorded as the end point; performing full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block; calculating the probability of traversing all segmentation methods for each semantic block in a left-to-right order, and selecting the segmentation method with the highest probability as the final segmentation result. The segmentation scheme of this invention is based on probability, traverses all solutions within a block, and comprehensively considers the context of the text, resulting in more accurate segmentation results, reducing labor costs, and improving segmentation accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a text segmentation method, system, computer device, and storage medium. Background Technology

[0002] In the field of Natural Language Processing (NLP), the use of language model pre-training methods has achieved significant improvements on many NLP tasks and has attracted widespread attention.

[0003] Currently, NLP pre-trained models construct a vocabulary using Byte Pair Encoding (BPE) and then perform word segmentation using the maximum forward matching algorithm. This word segmentation method leads to the following problems:

[0004] 1. The vocabulary generated by the BPE method only considers word frequency and ignores semantic considerations, resulting in inaccurate word segmentation results.

[0005] 2. Maximum forward matching segmentation only pursues the optimal local segmentation solution with greediness, without global consideration. It cannot represent the most semantics with the smallest segmentation block, resulting in a waste of resources. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a text segmentation method, system, computer device and storage medium to solve the problems of inaccurate segmentation results of NLP pre-trained models in the prior art, the inability to represent the most semantics with the smallest segmentation block, and the resulting waste of resources.

[0007] A first aspect of the present invention provides a method for word segmentation of text, comprising: acquiring text to be segmented;

[0008] For each sentence in the text to be segmented, the sentence is divided into multiple semantic blocks in order from left to right, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point.

[0009] Perform a full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block;

[0010] For each semantic block, the probability of all possible segmentation methods is calculated by traversing from left to right, and the segmentation method with the highest probability is selected as the final word segmentation result.

[0011] The vocabulary lists for each domain are pre-trained, and each domain's vocabulary list includes multiple words and the probability of each word appearing in that domain.

[0012] In one embodiment of the present invention, the step of training vocabulary lists for various domains includes:

[0013] Acquire corpora from different fields;

[0014] For each sentence in the corpus of each domain, determine whether two adjacent characters have a connected record in the preset vocabulary, and record multiple consecutive characters with connected records as a word;

[0015] Generate a candidate word list for each domain based on each word;

[0016] The probability of occurrence of each word in each candidate word list is calculated iteratively. In each iteration, words with a probability of occurrence less than the first threshold are discarded. The candidate word lists obtained after the iteration are used as word lists for the corresponding domain.

[0017] In one embodiment of the present invention, the step of determining whether two adjacent characters have consecutive records in a preset vocabulary, and recording multiple consecutive characters with consecutive records as a single word, includes:

[0018] Using the BERT model, the system sequentially checks whether two adjacent characters have a connected record in a pre-defined vocabulary, from left to right. If a connected record exists, the two adjacent characters are assigned the same identifier value; otherwise, the latter character is assigned a different identifier value than the former character.

[0019] Multiple consecutive characters with the same identifier value are recorded as a word.

[0020] In one embodiment of the present invention, the step of generating a candidate word list for each domain based on the vocabulary includes:

[0021] Calculate the probability of each word appearing in its respective domain, and select words whose probability of appearance is greater than the second threshold, along with their corresponding probability of appearance, to add to the candidate word list for each domain.

[0022] In one embodiment of the present invention, the step of acquiring corpora from different domains includes:

[0023] We used Scrapy crawlers to collect corpora for each domain;

[0024] After cleaning the collected corpora from each domain, they are then broken down into individual sentences.

[0025] In one embodiment of the present invention, the preset vocabulary is constructed based on corpora from different fields, NLP corpora, and general dictionaries.

[0026] In one embodiment of the present invention, the vocabulary length of each semantic block is less than or equal to 15.

[0027] A second aspect of the present invention also provides a text segmentation system, comprising:

[0028] The text acquisition module is used to acquire the text to be segmented.

[0029] The text segmentation module is used to divide each sentence of the text to be segmented into multiple semantic blocks in a left-to-right order, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point. The vocabulary of each domain is pre-trained and includes multiple words and the probability of each word appearing in the domain.

[0030] The segmentation module performs a full segmentation on each semantic block, obtaining all possible segmentation methods for that semantic block. For each semantic block, it iterates through all possible segmentation methods from left to right, calculates the probability, and selects the segmentation method with the highest probability as the final word segmentation result.

[0031] A third aspect of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps described in the text segmentation method of the first aspect of the present invention.

[0032] A fourth aspect of the invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in a text segmentation method of the first aspect of the invention.

[0033] As described above, the text segmentation method, system, computer device, and storage medium of the present invention have the following beneficial effects:

[0034] This invention constructs a vocabulary based on corpora from multiple domains, covering a wider range of words. Using this vocabulary, users can select multiple segmentation schemes for downstream tasks. Simultaneously, based on the constructed vocabulary, this invention divides the text to be segmented into multiple semantic blocks with endpoints as boundaries. Then, based on probability, it iterates through all solutions within a block, selecting the segmentation method with the highest probability as the final segmentation result. This invention comprehensively considers the context of the text, avoiding prioritizing matching preceding words regardless of subsequent segmentation, thus achieving more accurate segmentation results, reducing labor costs, and improving segmentation accuracy. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 The diagram shown is a flowchart of the word segmentation method in the first embodiment of the present invention.

[0037] Figure 2 The diagram shows a flowchart of the process of training and obtaining a vocabulary in the first embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram illustrating the process of obtaining vocabulary in the first embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram illustrating the process of generating a subsequent vocabulary in the first embodiment of the present invention.

[0040] Figure 5 The diagram shown is a structural block diagram of the word segmentation system in the second embodiment of the present invention.

[0041] Figure 6 The diagram shown is a schematic representation of a computer device according to a third embodiment of the present invention. Detailed Implementation

[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the figures only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0045] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0046] Please see Figure 1 The first embodiment of the present invention relates to a text segmentation method, which is applied to a text pre-training model for recognizing segmented text. Specifically:

[0047] Step 101: Obtain the text to be segmented.

[0048] Specifically, before segmenting the text, it should be preprocessed. The preprocessing steps may include any one or more of the following:

[0049] Step 1: Clean the text to be segmented, filtering out noisy data and retaining useful data. Cleaning methods may include: manual deduplication, alignment, deletion, and annotation. In addition, regular expression matching can be used to remove redundant punctuation and web page tags from the corpus.

[0050] Step 2: Convert the traditional Chinese characters in the text to be segmented into simplified Chinese characters. For example, you can convert the traditional Chinese characters in the text to be segmented into simplified Chinese characters based on public data sources or traditional and simplified Chinese character libraries, thereby reducing the parameter space size of the text pre-training model.

[0051] Step 3: Split the text to be segmented into individual sentences. For example, punctuation marks such as commas, periods, and question marks can be used to split the text into multiple individual sentences. In a feasible embodiment, the built-in functions of the Java language can be used to perform the function of splitting individual sentences.

[0052] It should be noted that the three preprocessing steps described above can be flexibly combined and used according to the actual situation. For example, when the text to be segmented is in English, step two is unnecessary, and only steps one and three need to be performed. In addition, the preprocessing of this application is not limited to the above three steps, and may also include other steps, such as synonym or near-synonym replacement processing, etc., which are not limited in this application.

[0053] Step 102: For each sentence in the text, divide the sentence into multiple semantic blocks in order from left to right, with the end point as the boundary.

[0054] Specifically, this embodiment uses the BERT model to determine whether each character is an end point from left to right. If it is an end point, the end point and the multiple consecutive characters preceding it are recorded as a semantic block. An end point is defined as follows: if two consecutive characters do not have a connected record in the vocabulary of the text to be segmented, then the character preceding the two consecutive characters is recorded as the end point. For example, in the sentence "Annuity insurance refers to the policyholder or insured paying premiums in a lump sum or on a regular basis," the BERT model checks for the existence of an end point for each word in the sentence, proceeding from left to right. If "annuity" and "annuity" have a connected record in the insurance vocabulary, it then checks for a connected record between "annuity" and "insurance." If it determines that "insurance" and "is" do not have a connected record, it considers "insurance" to be an end point in the sentence, and records the four words "annuity insurance" as a semantic block. This process continues until the semantic blocks of "annuity insurance," "is," "refers to," "policyholder," "or," "insured," "once," "or," "on a regular basis," "pay," and "premium" are obtained.

[0055] It should be understood that the BERT model is a language encoder. The large-scale corpus on which the BERT model is based covers various knowledge domains. The pre-trained language model trained on such corpus can be used to solve natural language problems in different domains. The BERT model introduces two pre-training tasks: MLM (Masked Language Model) and NSP (NextSentence Prediction). First, the language model is pre-trained on a large amount of unsupervised corpus, and then fine-tuned using a small amount of labeled corpus to complete downstream NLP tasks such as text classification, sequence labeling, machine translation, and reading comprehension. In this embodiment, the open-source BERT model can be used, or it can be trained on the basis of the open-source BERT model according to the specific task.

[0056] To elaborate further, the vocabulary lists for each domain are pre-trained, and each domain's vocabulary list includes multiple words and the probability of each word appearing in that domain.

[0057] Please see Figure 2 The steps for training vocabulary lists for various domains include:

[0058] Step 201: Obtain corpora from different fields.

[0059] Specifically, existing pre-trained text models typically use vocabularys built from publicly available NLP corpora. However, NLP corpora generally only cover general domains, neglecting vertical domains such as finance and novels. For tasks in more refined domains, the lack of corresponding entity words leads to poor word segmentation performance in pre-trained text models. To expand the vocabulary and enable users to select segmentation schemes for specific downstream tasks, this embodiment uses the Scrapy tool to crawl web page data, thereby obtaining corpora from multiple different domains. It should be understood that before using Scrapy to obtain corpora, a master server and at least one slave server are pre-set. The master server maintains the task distribution queue and stores the data crawled by the Scrapy tool. The master server is pre-configured with relevant information to be crawled, corresponding website source parameters, website source links, and crawler task queues. Each slave server runs Scrapy scripts to execute crawler tasks. When the Scrapy tool receives the assigned task queue, it initiates access requests to the target website source according to the task queue, crawls the required pages, and after parsing the web pages, returns the crawled data to the master server's database for storage. In addition, the scrapy tool returns a list of completed tasks to the main server for task deduplication and scheduling to prevent duplicate web pages.

[0060] Using the above approach, this embodiment collected a total of 100GB of corpus data from multiple websites with a large number of users, including Zhihu, research reports, Baidu Encyclopedia, Tieba, Douban, Hupu, and Biquge, covering five fields: finance, novels, law, news, and sports.

[0061] To elaborate further, after acquiring corpora from multiple domains, it is necessary to clean the acquired corpora, filter out noisy data, and retain useful data. Cleaning methods may include: manual deduplication, alignment, deletion, and annotation. In addition, regular expression matching can be used to remove redundant punctuation and web page tags from the corpus.

[0062] The cleaned corpus is then split into individual sentences. The splitting method can utilize punctuation marks, such as commas, periods, and question marks, to divide the corpus into multiple individual sentences. In a feasible embodiment, the built-in functions of the Java language can also be used to perform the function of splitting individual sentences.

[0063] Step 202: For each sentence in the corpus of each field, determine whether there is a connected record in the preset word list for two adjacent characters, and record multiple consecutive characters with connected records as one vocabulary; generate a candidate word list for each field according to each vocabulary.

[0064] Specifically, since the candidate word list in this embodiment serves the NLP pre-training model, in order to maintain the consistency between the vocabulary and the model, this embodiment preferentially uses the NLP model to define the candidate word list.

[0065] Please refer to Figure 3 , the steps of determining whether there is a connected record in the preset word list for two adjacent characters and recording multiple consecutive characters with connected records as one vocabulary include:

[0066] Step 301: Use the BERT model to sequentially determine whether there is a connected record in the preset word list for two adjacent characters in the order from left to right;

[0067] Step 302: If there is a connected record, assign the same identification value to the two adjacent characters; otherwise, assign a different identification value to the latter character from the former character;

[0068] Step 303: Record multiple consecutive characters with the same identification value as one vocabulary.

[0069] For example, for the sentence "Annuity insurance means that the applicant or the insured pays the insurance premium once or periodically", the BERT model sequentially determines whether there is a connected record in the preset word list between two adjacent characters in the order from left to right and outputs an identification value. When the new character and the previous character belong to the same semantic block, the old value is still output. For example, if there is a connected record in the preset word list for every two consecutive characters of "annuity insurance", then each character of "annuity insurance" is marked as 1; when there is no connected record in the preset word list between the "insurance" of "annuity insurance" and "is", a new value is output at this time, changing from 1 to 0; similarly, there is no connected record in the preset word list between "is" and "means", so "is" is marked as the end point, and at this time, an identification value different from the previous one is output, changing from 0 to 1; finally, the output identification value is "1 1 1 1 00 1 1 1 0 1 1 1 1 0 0 1 0 0 1 1 0 0 0". Among them, the consecutive identification values "1 1 1 1" are considered as one vocabulary, and at the same time, the consecutive identification values "0 0" are also considered as one vocabulary.

[0070] It should be noted that the preset vocabulary in this embodiment is composed of any one or more of the corpora collected in step 201 for each domain, publicly available NLP corpora, and general dictionaries.

[0071] Please see Figure 4 The steps for generating a candidate vocabulary for each domain based on each word include:

[0072] Step 401: Calculate the probability of each word appearing in its respective domain;

[0073] Step 402: Determine whether the probability of occurrence of each word is greater than the second threshold;

[0074] Step 403: If the probability of a word appearing is greater than the second threshold, then add the word and its corresponding probability of appearance to the candidate word list.

[0075] Step 404: If the probability of a word appearing is less than the second threshold, it is considered that the information content of the word is too small. In order to reduce the redundancy of the candidate word list, the word should be discarded and the number of words in the candidate word list should be maintained within a reasonable range.

[0076] Specifically, this embodiment uses the sentencepiece model to calculate the probability of each word in each domain, obtain the probability of each word appearing in that domain, and select words with higher occurrence probabilities and their occurrence probabilities to add to the candidate word list.

[0077] It should be understood that the SentencePiece model is a word segmentation model. For a given initial vocabulary, the SentencePiece model uses the EM algorithm to iteratively calculate the occurrence probability of all words in the initial vocabulary. For each word, after each iteration, it calculates the loss caused by its removal and removes words with smaller losses, thereby achieving better word segmentation results.

[0078] Furthermore, the candidate word list composed of each word in this embodiment is an initial screening of the corpus in each field. It should be understood that, in order to adapt to different downstream tasks, the candidate word list may pre-store general words corresponding to different downstream tasks. Each general word and the words selected in this embodiment together constitute the candidate word list.

[0079] Step 203: Iteratively calculate the occurrence probability of each word in each candidate word list, and in each iteration, discard words whose occurrence probability is less than the first threshold. The candidate word lists obtained after the iteration are used as word lists for the corresponding domains.

[0080] Specifically, the sentencepiece model is used to iteratively calculate the probability of occurrence of each word in each candidate vocabulary list. In each iteration, words with a probability of occurrence less than a first threshold are discarded. The iteration ends when the preset number of iterations is reached, and the candidate vocabulary list obtained at this point is taken as the vocabulary list for that domain. It should be understood that the vocabulary list for each domain includes multiple words and the probability of occurrence of each word in that domain.

[0081] Through steps 201-203 above, in this embodiment, for each domain, a vocabulary of 50,000 words and the probability of occurrence of each word in the vocabulary can be obtained.

[0082] Step 103: Perform full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block. For each semantic block, calculate the probability of all possible segmentation methods by traversing them from left to right, and select the segmentation method with the highest probability as the final word segmentation result.

[0083] Specifically, the granularity of word segmentation has a significant impact on the segmentation results. If the granularity is too large, it carries too much information and is difficult to reuse; if the granularity is too small, it will lead to excessive computational complexity. Therefore, in order to improve the accuracy of word segmentation, this embodiment obtains multiple semantic blocks with slightly larger granularity through the method in step 102. Then, based on each semantic block, each semantic block is fully segmented to obtain all possible segmentation methods for that semantic block.

[0084] For example, for the sentence "Xiaoming reports the contents of the book to the teacher", according to the method in step 102, the sentence can be divided into several semantic blocks: "Xiaoming", "to", "teacher", and "report the contents of the book". In step 103, based on the word list obtained in step 203, the semantic block "report the contents of the book" is further fully segmented. Following the order from left to right, each character in the semantic block is traversed in turn to find all words starting with that character, thus obtaining all the segmentation methods of the semantic block.

[0085] For all segmentation methods, the probabilities are calculated by traversing them from left to right, and the segmentation method with the highest probability is selected as the final word segmentation result. For example, for a semantic block C, there are multiple segmentation methods S, and the expression for the probability P(S|C) of each segmentation method is:

[0086]

[0087] Where P(C) is the probability of semantic block C appearing in the vocabulary of its domain, which is a fixed value; P(C|S) is a fixed value of 1.

[0088] P(S) = P(W1, W2, ... W) n)≈P(W1)×P(W2)×...×P(W n )

[0089] ∝logP(W1)+logP(W2)+…+logP(W n )

[0090] Among them, P(W n Let be the probability of each word appearing in the vocabulary of the semantic block's domain for each segmentation method. Using the above method, the probabilities of all segmentation methods for the semantic block are calculated sequentially, and the segmentation method with the highest probability is selected as the final word segmentation result.

[0091] It should be noted that, in order to avoid excessively long semantic blocks leading to high computational complexity, this embodiment limits the maximum vocabulary length of each semantic block to less than or equal to 15. If the maximum vocabulary length is exceeded, it will be forcibly truncated.

[0092] As can be seen, this implementation constructs a vocabulary based on corpora from multiple domains, covering a wider range of words. Using this vocabulary, users can select various segmentation schemes for downstream tasks. Simultaneously, based on the constructed vocabulary, this invention divides the text to be segmented into multiple semantic blocks with the end point as the boundary. Then, based on probability, it iterates through all solutions within a block, selecting the segmentation method with the highest probability as the final segmentation result. This invention comprehensively considers the context of the text, avoiding prioritizing matching preceding words regardless of subsequent segmentation, thus achieving more accurate segmentation results, reducing labor costs, and improving segmentation accuracy.

[0093] Please see Figure 5 The second embodiment of the present invention relates to a text segmentation system, specifically including:

[0094] The text acquisition module is used to acquire the text to be segmented.

[0095] Specifically, before segmenting the text, it should be preprocessed. The preprocessing steps may include any one or more of the following:

[0096] Step 1: Clean the text to be segmented, filtering out noisy data and retaining useful data. Cleaning methods may include: manual deduplication, alignment, deletion, and annotation. In addition, regular expression matching can be used to remove redundant punctuation and web page tags from the corpus.

[0097] Step 2: Convert the traditional Chinese characters in the text to be segmented into simplified Chinese characters. For example, you can convert the traditional Chinese characters in the text to be segmented into simplified Chinese characters based on public data sources or traditional and simplified Chinese character libraries, thereby reducing the parameter space size of the text pre-training model.

[0098] Step 3: Split the text to be segmented into individual sentences. For example, punctuation marks such as commas, periods, and question marks can be used to split the text into multiple individual sentences. In a feasible embodiment, the built-in functions of the Java language can be used to perform the function of splitting individual sentences.

[0099] The text segmentation module is used to divide each sentence of the text to be segmented into multiple semantic blocks in a left-to-right order, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point. The vocabulary of each domain is pre-trained and includes multiple words and the probability of each word appearing in that domain.

[0100] Specifically, this embodiment uses the BERT model to determine whether each character is an end point from left to right. If it is an end point, the end point and the multiple consecutive characters preceding it are recorded as a semantic block. An end point is defined as follows: if two consecutive characters do not have a connected record in the vocabulary of the text to be segmented, then the character preceding the two consecutive characters is recorded as the end point.

[0101] The vocabulary lists for each domain are pre-trained and include multiple words and their probability of occurrence in that domain. The steps for training the vocabulary lists for each domain include:

[0102] Step 1: Acquire corpora from different fields;

[0103] Step 2: For each sentence in the corpus of each domain, determine whether there are consecutive records of two adjacent characters in the preset vocabulary. Record multiple consecutive characters with consecutive records as a word. Generate a candidate vocabulary for each domain based on each word.

[0104] Step 3: Iteratively calculate the occurrence probability of each word in each candidate word list, and discard words with an occurrence probability less than the first threshold in each iteration. Use the candidate word lists obtained after the iteration as the word lists of the corresponding domains.

[0105] Specifically, existing pre-trained text models typically use vocabularys built from publicly available NLP corpora. However, NLP corpora generally only cover general domains, neglecting vertical domains such as finance and novels. For tasks in more refined domains, the lack of corresponding entity words leads to poor word segmentation performance in pre-trained text models. To expand the vocabulary and enable users to select segmentation schemes for specific downstream tasks, this embodiment uses the Scrapy tool to crawl web page data, thereby obtaining corpora from multiple different domains. It should be understood that before using Scrapy to obtain corpora, a master server and at least one slave server are pre-set. The master server maintains the task distribution queue and stores the data crawled by the Scrapy tool. The master server is pre-configured with relevant information to be crawled, corresponding website source parameters, website source links, and crawler task queues. Each slave server runs Scrapy scripts to execute crawler tasks. When the Scrapy tool receives the assigned task queue, it initiates access requests to the target website source according to the task queue, crawls the required pages, and after parsing the web pages, returns the crawled data to the master server's database for storage. In addition, the scrapy tool returns a list of completed tasks to the main server for task deduplication and scheduling to prevent duplicate web pages.

[0106] Using the above approach, this embodiment collected a total of 100GB of corpus data from multiple websites with a large number of users, including Zhihu, research reports, Baidu Encyclopedia, Tieba, Douban, Hupu, and Biquge, covering five fields: finance, novels, law, news, and sports.

[0107] To elaborate further, after acquiring corpora from multiple domains, it is necessary to clean the acquired corpora, filter out noisy data, and retain useful data. Cleaning methods may include: manual deduplication, alignment, deletion, and annotation. In addition, regular expression matching can be used to remove redundant punctuation and web page tags from the corpus.

[0108] The cleaned corpus is then split into individual sentences. The splitting method can utilize punctuation marks, such as commas, periods, and question marks, to divide the corpus into multiple individual sentences. In a feasible embodiment, the built-in functions of the Java language can also be used to perform the function of splitting individual sentences.

[0109] To further explain, the steps for determining whether two adjacent characters have consecutive records in a preset vocabulary, and recording multiple consecutive characters with consecutive records as a single word, include:

[0110] Using the BERT model, the system sequentially checks whether two adjacent characters have a connected record in a pre-defined vocabulary, from left to right. If a connected record exists, the two adjacent characters are assigned the same identifier value; otherwise, the latter character is assigned a different identifier value than the former character. Multiple consecutive characters with the same identifier value are recorded as a word.

[0111] The steps for generating a candidate vocabulary list for each domain based on each word include:

[0112] The sentencepiece model is used to iterate and calculate the probability of each word in its domain. It then determines whether the probability of each word is greater than a second threshold. If the probability of a word is greater than the second threshold, the word and its corresponding probability are added to the candidate word list. If the probability of a word is less than the second threshold, the word is considered to have too little information. To reduce the redundancy of the candidate word list, the word should be discarded, and the number of words in the candidate word list should be kept within a reasonable range.

[0113] Furthermore, the candidate word list composed of each word in this embodiment is an initial screening of the corpus in each field. It should be understood that, in order to adapt to different downstream tasks, the candidate word list may pre-store general words corresponding to different downstream tasks. Each general word and the words selected in this embodiment together constitute the candidate word list.

[0114] To elaborate further, the sentencepiece model is used to iteratively calculate the probability of occurrence of each word in each candidate vocabulary list. In each iteration, words with a probability of occurrence less than a first threshold are discarded. The iteration ends when the preset number of iterations is reached, and the candidate vocabulary list obtained at this point is taken as the vocabulary list for that domain. It should be understood that the vocabulary list for each domain includes multiple words and the probability of occurrence of each word in that domain.

[0115] Through the above steps, for each domain, a vocabulary of 50,000 words and the probability of occurrence of each word in the vocabulary can be obtained.

[0116] The segmentation module performs a full segmentation on each semantic block, obtaining all possible segmentation methods for that semantic block. For each semantic block, it iterates through all possible segmentation methods from left to right, calculates the probability, and selects the segmentation method with the highest probability as the final word segmentation result.

[0117] Specifically, the granularity of word segmentation significantly impacts the segmentation results. Too large a granularity carries excessive information, making it difficult to reuse; too small a granularity leads to excessive computational complexity. Therefore, to improve segmentation accuracy, this embodiment uses a text segmentation module to obtain multiple semantic blocks with slightly larger granularity. Then, based on each semantic block, a full segmentation is performed. Following a left-to-right order, each character in the semantic block is traversed sequentially to identify all words starting with that character, resulting in all possible segmentation methods for that semantic block. For all segmentation methods, the probability is calculated by traversing them from left to right, and the segmentation method with the highest probability is selected as the final segmentation result.

[0118] As can be seen, this implementation constructs a vocabulary based on corpora from multiple domains, covering a wider range of words. Using this vocabulary, users can select various segmentation schemes for downstream tasks. Simultaneously, based on the constructed vocabulary, this invention divides the text to be segmented into multiple semantic blocks with the end point as the boundary. Then, based on probability, it iterates through all solutions within a block, selecting the segmentation method with the highest probability as the final segmentation result. This invention comprehensively considers the context of the text, avoiding prioritizing matching preceding words regardless of subsequent segmentation, thus achieving more accurate segmentation results, reducing labor costs, and improving segmentation accuracy.

[0119] Please see Figure 6 The third embodiment of the present invention relates to a computer device, including a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor 602. When the processor 602 executes the computer program, it performs the following steps:

[0120] Get the text to be segmented;

[0121] For each sentence in the text, the sentence is divided into multiple semantic blocks in order from left to right, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point.

[0122] Perform a full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block;

[0123] For each semantic block, the probability of all possible segmentation methods is calculated by traversing from left to right, and the segmentation method with the highest probability is selected as the final word segmentation result.

[0124] The vocabulary lists for each domain are pre-trained, and each domain's vocabulary list includes multiple words and the probability of each word appearing in that domain.

[0125] The memory 601 and processor 602 are connected via a bus. This bus can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 602 and memory 601. The bus can also connect various other circuits, such as peripheral devices 603, voltage regulators 604, and power management circuits, which are well-known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 602 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 602.

[0126] Processor 602 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 601 can be used to store data used by processor 602 during operation.

[0127] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0128] The fifth embodiment of the present invention relates to a storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0129] Get the text to be segmented;

[0130] For each sentence in the text, the sentence is divided into multiple semantic blocks in order from left to right, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point.

[0131] Perform a full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block;

[0132] For each semantic block, the probability of all possible segmentation methods is calculated by traversing from left to right, and the segmentation method with the highest probability is selected as the final word segmentation result.

[0133] The vocabulary lists for each domain are pre-trained, and each domain's vocabulary list includes multiple words and the probability of each word appearing in that domain.

[0134] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0135] In summary, the text segmentation method, system, computer device, and storage medium of this invention construct a vocabulary based on corpora from multiple domains, covering a wider range of words. Using this vocabulary, users can select multiple segmentation schemes for downstream tasks. Furthermore, based on the constructed vocabulary, this invention divides the text to be segmented into multiple semantic blocks with the end point as the boundary. Based on probability, it iterates through all solutions within a block, selecting the segmentation method with the highest probability as the final segmentation result. This invention comprehensively considers the context of the text, avoiding prioritizing matching preceding words regardless of subsequent segmentation, thus achieving more accurate segmentation results, reducing labor costs, and improving segmentation accuracy. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0136] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A text segmentation method, characterized in that, include: Get the text to be segmented; For each sentence of the text, the sentence is divided into multiple semantic blocks in order from left to right, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point. Perform a full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block; For each semantic block, the probability of all possible segmentation methods is calculated by traversing from left to right, and the segmentation method with the highest probability is selected as the final word segmentation result. The expression for the probability P(S|C) of each segmentation method is: Where P(C) is the probability of semantic block C appearing in the vocabulary of its domain, which is a fixed value; P(C|S) is a fixed value of 1. P(S)=P(W1,W2,…W n )≈P(W1)×P(W2)×…×P(W n )∝logP(W1)+logP(W2)+…+logP(W n ); Among them, P(W n ) represents the probability of each word appearing in the vocabulary of the semantic block's domain in each segmentation method; The vocabulary lists for each domain are pre-trained, and each domain's vocabulary list includes multiple words and the probability of each word appearing in that domain.

2. The word segmentation method according to claim 1, characterized in that, The steps for training vocabularies for various domains include: Acquire corpora from different fields; For each sentence in the corpus of each domain, determine whether two adjacent characters have a connected record in the preset vocabulary, and record multiple consecutive characters with connected records as a word; Generate a candidate word list for each domain based on each word; The probability of occurrence of each word in each candidate word list is calculated iteratively. In each iteration, words with a probability of occurrence less than the first threshold are discarded. The candidate word lists obtained after the iteration are used as word lists for the corresponding domain.

3. The word segmentation method according to claim 2, characterized in that: The step of determining whether two adjacent characters have a connected record in a preset vocabulary, and recording multiple consecutive characters with connected records as a single word, includes: Using the BERT model, the system sequentially checks whether two adjacent characters have a connected record in a pre-defined vocabulary, from left to right. If a connected record exists, the two adjacent characters are assigned the same identifier value; otherwise, the latter character is assigned a different identifier value than the former character. Multiple consecutive characters with the same identifier value are recorded as a word.

4. The word segmentation method according to claim 2, characterized in that, The step of generating a candidate word list for each domain based on each of the aforementioned words includes: Calculate the probability of each word appearing in its respective domain, and select words whose probability of appearance is greater than the second threshold, along with their corresponding probability of appearance, to add to the candidate word list for each domain.

5. The word segmentation method according to claim 2, characterized in that, The steps for acquiring corpora from different domains include: We used Scrapy crawlers to collect corpora for each domain; After cleaning the collected corpora from each domain, they are then broken down into individual sentences.

6. The word segmentation method according to claim 5, characterized in that, The preset vocabulary is composed of any one or more corpora from different fields, NLP corpora, and general dictionaries.

7. The text segmentation method according to claim 1, characterized in that, Each semantic block has a vocabulary length of 15 or less.

8. A text segmentation system, characterized in that, include: The text acquisition module is used to acquire the text to be segmented. The text segmentation module is used to divide each sentence of the text to be segmented into multiple semantic blocks in a left-to-right order, with the end point as the boundary. If two consecutive characters do not have a connected record in the vocabulary of the domain to which the text to be segmented belongs, then the preceding character of the two consecutive characters is recorded as the end point. The vocabulary of each domain is pre-trained and includes multiple words and the probability of each word appearing in the domain. The segmentation module performs a full segmentation on each semantic block, obtaining all possible segmentation methods for that semantic block. For each semantic block, it iterates through all possible segmentation methods from left to right, calculates the probability, and selects the segmentation method with the highest probability as the final word segmentation result. The expression for the probability P(S|C) of each segmentation method is: Where P(C) is the probability of semantic block C appearing in the vocabulary of its domain, which is a fixed value; P(C|S) is a fixed value of 1. P(S)=P(W1,W2,…W n )≈P(W1)×P(W2)×…×P(W n )∝logP(W1)+logP(W2)+…+logP(W n ); Among them, P(W n ) represents the probability of each word appearing in the vocabulary of the semantic block's domain in each segmentation method.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the text segmentation method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the text segmentation method according to any one of claims 1-7.

Images