A text segmentation method, system, computer device, and storage medium

By constructing a multi-domain lexicon and selecting the segmentation method with the highest probability within a semantic block, the word segmentation method solves the problems of inaccurate word segmentation and resource waste in existing technologies, achieving more accurate word segmentation results and resource optimization.

CN114417854BActive Publication Date: 2025-12-02ONE CONNECT SMART TECH CO LTD SHENZHEN
2 Cites 0 Cited by

Patent Information

Application Number
CN202111568385.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-17
Publication Date
2025-12-02
Estimated Expiration
2041-12-17

AI Technical Summary

Technical Problem

Existing NLP pre-trained models neglect the semantic level in their word segmentation methods, resulting in inaccurate segmentation results. Furthermore, the maximum forward matching algorithm cannot represent the most semantics with the smallest segmentation block, leading to a waste of resources.

Method used

By constructing a vocabulary based on multiple domains, using the BERT model to determine the end point of semantic blocks, the text is divided into multiple semantic blocks. Within each semantic block, all possible segmentation methods are traversed, and the one with the highest probability is selected as the final word segmentation result.

Benefits of technology

It improves word segmentation accuracy, reduces labor costs, comprehensively considers the context of the text, and avoids resource waste.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

This invention relates to the field of artificial intelligence technology, providing a text segmentation method, system, computer device, and storage medium, comprising: acquiring the text to be segmented; dividing each sentence of the text into multiple semantic blocks according to a left-to-right order, with the end point as the boundary; if two consecutive characters do not have a connected record in the vocabulary of the text to be segmented, then the preceding character of the two consecutive characters is recorded as the end point; performing full segmentation on each semantic block to obtain all possible segmentation methods for that semantic block; calculating the probability of traversing all segmentation methods for each semantic block in a left-to-right order, and selecting the segmentation method with the highest probability as the final segmentation result. The segmentation scheme of this invention is based on probability, traverses all solutions within a block, and comprehensively considers the context of the text, resulting in more accurate segmentation results, reducing labor costs, and improving segmentation accuracy.
Need to check novelty before this filing date? Find Prior Art