Word segmentation method and apparatus, and non-volatile storage medium and electronic device

Abstract

Disclosed in the present invention are a word segmentation method and apparatus, and a non-volatile storage medium and an electronic device. The method comprises: acquiring text to be subjected to word segmentation that comprises a plurality of target words; on the basis of a preset vocabulary, adding a corresponding target label for each target word, so as to obtain a target label queue, wherein the target label queue comprises a plurality of target labels arranged according to a target sequence, the target sequence being an arrangement sequence of the target words corresponding to the target labels in said text; on the basis of a pre-configured target word segmentation rule in a preset label dictionary, splitting the target label queue into a plurality of target label sub-queues; and converting target labels in each target label sub-queue into corresponding target words, so as to obtain target word-segmented text.

Description

Word segmentation method, device, non-volatile storage medium and electronic device

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2024101401845, filed on January 31, 2024, entitled “Word segmentation method, device, non-volatile storage medium and electronic device”, the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present invention relates to the field of machine learning, and in particular to a word segmentation method, device, non-volatile storage medium, electronic device, and computer program product. Background Art

[0004] Traditional intelligent customer service scenarios typically employ FAQs (Frequently Asked Questions) or KBQA (Knowledge-Based Question Answering), focusing on simple questions and single intents. However, there are often no established technical solutions for understanding complex sentences. Historically, training multi-intent models has been used to address this problem, but this approach has limitations and can only be used in specific scenarios, lacking portability.

[0005] Summary of the Invention

[0006] Embodiments of the present invention provide a word segmentation method, device, non-volatile storage medium, electronic device, and computer program product.

[0007] According to one aspect of an embodiment of the present invention, a word segmentation method is provided, comprising: obtaining a text to be segmented, wherein the text to be segmented includes: a plurality of target words; adding a corresponding target tag to each of the target words according to a preset word table to obtain a target tag queue, wherein the preset word table is used to record a plurality of preset words and a preset tag corresponding to each of the preset words, the target tag is a preset tag corresponding to the preset word that matches the target word, and the target tag queue includes: a plurality of target tags arranged in a target order, wherein the target order is the order in which the target words corresponding to each target tag are arranged in the text to be segmented; according to a preset tag dictionary, A pre-configured target word segmentation rule splits the target tag queue into multiple target sub-tag queues, wherein the preset tag dictionary record is used to record multiple preset tag queues and the preset word segmentation rule corresponding to each preset tag queue, and the target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue. The preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that some of the preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues; the target tag in each target sub-tag queue is converted into the corresponding target word to obtain a target word segmentation text.

[0008] Optionally, according to the target word segmentation rules pre-configured in the preset tag dictionary, splitting the target tag queue into multiple target sub-tag queues includes: querying whether the preset tag queue that matches the target tag queue exists in the preset tag dictionary; if the preset tag queue that matches the target tag queue does not exist in the preset tag dictionary, using a preset word segmentation model to segment the text to be segmented to obtain a target word segmentation text; if the preset tag queue that matches the target tag queue exists in the preset tag dictionary, determining the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue as the target word segmentation rule, and splitting the target tag queue into multiple target sub-tag queues according to the target word segmentation rule.

[0009] Optionally, querying whether there is a preset tag queue matching the target tag queue in the preset tag dictionary includes: extracting the target tag arranged in the first position from the target tag queue as the first target tag; traversing the preset tag dictionary to see whether there is a first preset tag queue with the first target tag in the first position; if the first preset tag queue exists in the preset tag dictionary, placing the first preset tag queue into a first preset tag queue set; if the first preset tag queue does not exist in the preset tag dictionary, determining that there is no preset tag queue matching the target tag queue in the preset tag dictionary.

[0010] Optionally, when the first preset tag queue exists in the preset tag dictionary, after the first preset tag queue is placed into the first preset tag queue set, the method further includes: extracting the target tag arranged in the second position from the target tag queue as a second target tag, wherein the second position is located after the first position and adjacent to the first position; traversing the first preset tag queue set to see whether there is a second preset tag queue with the second position as the second target tag; when the second preset tag queue exists in the first preset tag queue set, placing the second preset tag queue into the second preset tag queue set, and taking the second position as the first position, and taking the second preset tag queue set as the first preset tag queue set; when the second preset tag queue does not exist in the first preset tag queue set, determining that the preset tag queue that matches the target tag queue does not exist in the preset tag dictionary.

[0011] Optionally, before extracting the target tag arranged in the second position from the target tag queue as the second target tag, the method also includes: detecting whether the target tag exists in the second position of the target tag queue; when the target tag does not exist in the second position of the target tag queue, traversing the first preset tag queue set to see whether there is a third preset tag queue in which the target tag does not exist in the second position; when the third preset tag queue exists in the first preset tag queue set, determining the third preset tag queue as the preset tag queue that matches the target tag queue; when the third preset tag queue does not exist in the first preset tag queue set, determining that the preset tag queue that matches the target tag queue does not exist in the preset tag dictionary.

[0012] Optionally, when there is no preset tag queue matching the target tag queue in the preset tag dictionary, segmenting the text to be segmented using a preset word segmentation model to obtain the target segmented text includes: segmenting the text to be segmented using a first word segmentation model to obtain a first segmented text, wherein the preset word segmentation model includes: the first word segmentation model, the first word segmentation model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: preset words, and part-of-speech tags of the preset words, the part-of-speech tags include: a first part-of-speech tag indicating that the preset word belongs to positive example data, and a second part-of-speech tag indicating that the preset word belongs to breakpoint data, the first word segmentation model is used to truncate the text to be segmented at the position of the target word belonging to the breakpoint data; when the first number of the first segmented texts is greater than a preset number threshold, obtaining the confidence of the first segmentation model; when the confidence is greater than a preset confidence threshold, determining the first segmented text as the target segmented text.

[0013] Optionally, after using the first word segmentation model to segment the text to be segmented to obtain the first word segmentation text, the method further includes: using the second word segmentation model to segment the text to be segmented according to semantics to obtain a second word segmentation text, wherein the second word segmentation model is used to determine the second word segmentation text corresponding to each semantic when the text to be segmented includes multiple semantics, and each second word segmentation text includes at least one target word; when the first number of the first word segmentation texts is not greater than the preset number threshold, and the second number of the second word segmentation texts is greater than the preset number threshold, using the second word segmentation model to determine whether the second number of the second word segmentation texts has the same target semantics as the text to be segmented; when the target semantics are the same, the second word segmentation text is determined as the target word segmentation text; when the target semantics are different, the first word segmentation text is determined as the target word segmentation text.

[0014] Optionally, after using the second word segmentation model to segment the text to be segmented according to semantics to obtain the second word segmentation text, the method further includes: judging whether the first quantity of the first word segmentation text is the same as the second quantity of the second word segmentation text; when the first quantity of the first word segmentation text is the same as the second quantity of the second word segmentation text, determining the second word segmentation text as the target word segmentation text; when the first quantity of the first word segmentation text is the same as the second quantity of the second word segmentation text, determining the preset word segmentation text with the least quantity as the target word segmentation text, wherein the preset word segmentation text includes the first word segmentation text and the second word segmentation text.

[0015] Optionally, the preset tag dictionary records multiple preset tag queues in a tree format.

[0016] Optionally, the preset word segmentation model is a model trained through machine learning.

[0017] Optionally, the first word segmentation model is a BERT model.

[0018] Optionally, the second word segmentation model is a large language model (LLM).

[0019] According to another aspect of an embodiment of the present invention, a word segmentation device is further provided, comprising: an acquisition module for acquiring a text to be segmented, wherein the text to be segmented includes: a plurality of target words; an adding module for adding a corresponding target tag for each of the target words according to a preset word table to obtain a target tag queue, wherein the preset word table is used to record a plurality of preset words and a preset tag corresponding to each of the preset words, the target tag is a preset tag corresponding to the preset word that matches the target word, and the target tag queue includes: a plurality of target tags arranged in a target order, wherein the target order is the arrangement order of the target words corresponding to each target tag in the text to be segmented; a splitting module for adding a corresponding target tag according to a preset word table. The target word segmentation rule is pre-configured in the preset tag dictionary, and the target tag queue is split into multiple target sub-tag queues, wherein the preset tag dictionary record is used to record multiple preset tag queues and the preset word segmentation rule corresponding to each preset tag queue, and the target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue. The preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that some of the preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues; a conversion module is used to convert the target tag in each target sub-tag queue into the corresponding target word to obtain a target word segmentation text.

[0020] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided. The non-volatile storage medium is used to store a program. When the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned word segmentation method.

[0021] According to another aspect of an embodiment of the present invention, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the processor, wherein the program executes the above-mentioned word segmentation method when running.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, wherein the computer program implements the steps of the above-mentioned word segmentation method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] FIG1 is a flow chart of a word segmentation method according to an embodiment of the present invention;

[0025] FIG2 is a schematic diagram of a complex sentence understanding process in an LLM-based intelligent customer service scenario according to an embodiment of the present invention;

[0026] FIG3 is a schematic diagram of constructing a preset tag dictionary according to an embodiment of the present invention;

[0027] FIG4 is a schematic diagram of an overall customer service question-and-answer process according to an embodiment of the present invention;

[0028] FIG5 is a schematic diagram of a word segmentation device according to an embodiment of the present invention;

[0029] FIG6 is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0033] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0034] FAQ: Frequently Asked Questions is a list of the most frequently asked questions about a topic or product and their answers. In a website, application, or service, the FAQ section provides answers to common user questions so that users can quickly find the information they need.

[0035] KBQA (Knowledge-Based Question Answering) is a technology that answers user questions by querying a knowledge base. The knowledge base contains structured information about a specific domain, and the KBQA system can obtain answers from it.

[0036] Trie: (also known as dictionary tree, prefix tree) A trie is a tree-like data structure used to store associative arrays, where the keys are usually strings. A trie is characterized by its ability to search, insert, or delete strings in O(L) time complexity, where L is the length of the string.

[0037] BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model based on the Transformer architecture that focuses on natural language processing tasks. BERT uses a large-scale corpus in the pre-training phase to learn contextual representations of language, and then fine-tunes it on specific tasks to improve its performance.

[0038] LLM: Large Language Model refers to a language model trained on large amounts of data. These models typically possess powerful natural language processing capabilities and can understand and generate natural language text.

[0039] In intelligent customer service scenarios, we need to understand and recognize complex user input scenarios. For example, when a user inputs "check traffic and last month's bill," "check traffic, weather, and phone balance," previous methods could only identify an executable command through a single intent, which lost some semantics. Alternatively, we could train a multi-task model to execute it, but this often has great limitations, lacks scenario migration, and is costly.

[0040] According to an embodiment of the present invention, a tokenization method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] FIG1 is a flow chart of a word segmentation method according to an embodiment of the present invention. As shown in FIG1 , the method includes the following steps:

[0042] Step S102, obtaining a text to be segmented including a plurality of target words;

[0043] Step S104: adding a corresponding target tag to each target word according to a preset vocabulary to obtain a target tag queue, wherein the preset vocabulary is used to record multiple preset words and preset tags corresponding to each preset word, the target tag is a preset tag corresponding to the preset word matching the target word, and the target tag queue includes: multiple target tags arranged in a target order, and the target order is the order in which the target words corresponding to each target tag are arranged in the text to be segmented;

[0044] Step S106: split the target tag queue into multiple target sub-tag queues according to the target word segmentation rule pre-configured in the preset tag dictionary, wherein the preset tag dictionary is used to record multiple preset tag queues and the preset word segmentation rule corresponding to each preset tag queue, the target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue matching the target tag queue, the preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that some preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues;

[0045] Step S108: convert the target tag in each target subtag queue into a corresponding target word to obtain a target word segmentation text.

[0046] In an embodiment of the present invention, a preset tag dictionary is used to record the correspondence between a preset tag queue and a preset word segmentation rule. In the process of segmenting the text to be segmented using the preset tag dictionary, the correspondence between the preset words and the preset tags recorded in the preset word list can be used to add target tags to multiple target words in the text to be segmented, thereby obtaining a target tag queue. Then, a preset tag queue matching the target tag queue is queried in the preset tag dictionary, and the preset word segmentation rule corresponding to the preset tag queue is used as the target word segmentation rule of the target tag queue. Then, the target word segmentation rule is used to select some target tags from the target tag queue for arrangement, thereby obtaining multiple target sub-tag queues, and each target sub-tag queue may include target tags corresponding to some target words in the text to be segmented. By converting the target tags in the target sub-tag queue into corresponding target words, the target segmentation text of the text to be segmented can be obtained, thereby achieving the technical effect of accurate word segmentation of the text to be segmented, thereby solving the technical problem that the existing technology cannot accurately segment complex sentences.

[0047] In the above step S104, the preset word table records the correspondence between a plurality of preset words and preset tags.

[0048] Optionally, the preset words corresponding to the preset tag "Search_action" include: {search, check, find, search, query, ...}; the preset words corresponding to the preset tag "Open_action" include: {open, open, ...}; the preset words corresponding to the preset tag "Close_action" include: {close, close, close, ...}; the preset words corresponding to the preset tag "Conjunction" include: {and, and, ...}; the preset words corresponding to the preset tag "Modal_action" include: {help me, for me, give me, ...}; the preset words corresponding to the preset tag "Stop_action" include: {please, excuse me, ...}; the preset tag "Entity" is used to introduce hanlp to identify noun vocabulary construction through part of speech.

[0049] In the above step S104, after adding a target tag to each target word in the text to be segmented, a target tag queue composed of target tags can be obtained, and the position of each target tag in the target tag queue is the same as the position of the target word corresponding to each target tag in the text to be segmented, thereby realizing the conversion of the text to be segmented into a target tag queue.

[0050] In the above step S106, the preset tag dictionary records the correspondence between multiple preset tag queues and preset word segmentation rules, wherein each preset tag queue includes multiple preset tags, and multiple preset tags are pre-placed at corresponding positions in the preset tag queue, that is, the preset tag queue is obtained by arranging multiple preset tags in sequence in advance, and the types of preset tags included in different preset tag queues may be different, the number of preset tags included in different preset tag queues may be different, and the arrangement order of the preset tags included in different preset tag queues may also be different.

[0051] In the above step S106 , the preset tag dictionary records multiple preset tag queues in a tree format, which can reduce the storage space required for storing the preset tag dictionary.

[0052] In the above step S106, the preset rule is used to extract some of the preset tags from the multiple preset tags of the corresponding preset tag queue for arrangement. The same preset tag queue can obtain multiple preset sub-tag queues using the preset rule, and there is at least one different preset tag in different preset sub-tag queues.

[0053] As an optional embodiment, according to the target word segmentation rules pre-configured in the preset tag dictionary, splitting the target tag queue into multiple target sub-tag queues includes: querying whether there is a preset tag queue matching the target tag queue in the preset tag dictionary; when there is no preset tag queue matching the target tag queue in the preset tag dictionary, using the preset word segmentation model to segment the text to be segmented to obtain the target word segmentation text; when there is a preset tag queue matching the target tag queue in the preset tag dictionary, determining the preset word segmentation rule corresponding to the preset tag queue matching the target tag queue as the target word segmentation rule, and splitting the target tag queue into multiple target sub-tag queues according to the target word segmentation rule.

[0054] In the above embodiment of the present invention, the most accurate word segmentation result can be obtained by performing word segmentation using the preset word segmentation rules pre-stored in the preset tag dictionary. However, if the preset tag dictionary does not contain a target word segmentation rule that matches the text to be segmented, the preset word segmentation model can be used to perform word segmentation to obtain a relatively accurate word segmentation result. Thus, the preset word segmentation model can be used to segment the text to be segmented for which no rules are recorded in the preset tag dictionary.

[0055] Optionally, the preset word segmentation model can be trained by machine learning.

[0056] As an optional embodiment, querying whether there is a preset tag queue matching the target tag queue in the preset tag dictionary includes: extracting the target tag arranged in the first position from the target tag queue as the first target tag; traversing the preset tag dictionary to see whether there is a first preset tag queue with the first target tag in the first position; if the first preset tag queue exists in the preset tag dictionary, placing the first preset tag queue into the first preset tag queue set; if the first preset tag queue does not exist in the preset tag dictionary, determining that there is no preset tag queue matching the target tag queue in the preset tag dictionary.

[0057] In the above embodiment of the present invention, when searching for a preset tag queue that matches a target tag queue in a preset tag dictionary, multiple target tags in the target tag queue can be queried in sequence according to the order of each target tag in the target tag queue, gradually reducing the number of preset tag queues that need to be queried, thereby improving query efficiency. In addition, when any target tag does not find the corresponding preset tag in the preset tag dictionary, it can be determined that the target word segmentation rule required by the target tag queue does not exist in the preset tag dictionary, and then jump to using the preset word segmentation model for word segmentation.

[0058] As an optional embodiment, when there is a first preset tag queue in the preset tag dictionary, after the first preset tag queue is placed into the first preset tag queue set, the method further includes: extracting the target tag arranged in the second position from the target tag queue as the second target tag, wherein the second position is located after the first position and adjacent to the first position; traversing the first preset tag queue set to see whether there is a second preset tag queue with the second target tag in the second position; when there is a second preset tag queue in the first preset tag queue set, placing the second preset tag queue into the second preset tag queue set, and taking the second position as the first position, and taking the second preset tag queue set as the first preset tag queue set; when there is no second preset tag queue in the first preset tag queue set, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0059] In the above embodiment of the present invention, when searching for a preset tag queue that matches a target tag queue in a preset tag dictionary, multiple target tags in the target tag queue can be queried in sequence according to the order of each target tag in the target tag queue, and the number of preset tag queues that need to be queried is gradually reduced until each target tag in the target tag queue has been queried. It is then determined that there is a preset tag queue including the target tag queue in the preset tag dictionary. If any target tag in the target tag queue does not find a corresponding preset tag in the preset tag dictionary, the query process is immediately terminated, and the preset word segmentation model is used for word segmentation, which can avoid invalid queries and improve word segmentation efficiency.

[0060] As an optional embodiment, before extracting the target tag arranged in the second position from the target tag queue as the second target tag, the method also includes: detecting whether there is a target tag in the second position of the target tag queue; when there is no target tag in the second position of the target tag queue, traversing the first preset tag queue set to see whether there is a third preset tag queue in which there is no target tag in the second position; when there is a third preset tag queue in the first preset tag queue set, determining the third preset tag queue as a preset tag queue that matches the target tag queue; when there is no third preset tag queue in the first preset tag queue set, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0061] In the above embodiment of the present invention, when searching for a preset tag queue that matches a target tag queue in a preset tag dictionary, it is necessary to ensure that the matching preset tag queue completely matches the target tag queue, wherein complete matching means that the number of preset tags in the preset tag queue is the same as the number of target tags in the target tag queue, the preset tags in the preset tag queue correspond one-to-one to the target tags in the target tag queue, and the position of each preset tag corresponding to the target tag in the preset tag queue is the same as the position of the target tag in the target tag queue. Therefore, when all target tags in the target tag queue have been queried, a preset tag queue with the same length as the target tag queue can be selected from the multiple preset tag queues screened out as the preset tag queue that matches the target tag queue, and the preset word segmentation rule corresponding to the preset tag queue is the target word segmentation rule of the target tag queue. If there is no preset tag queue with the same length as the target tag queue, it means that there is no preset tag queue that matches the target tag queue in the preset tag dictionary, and then the preset word segmentation model is used for word segmentation.

[0062] As an optional embodiment, when there is no preset tag queue matching the target tag queue in the preset tag dictionary, the preset word segmentation model is used to segment the text to be segmented, and obtaining the target word segmentation text includes: using a first word segmentation model to segment the text to be segmented to obtain a first word segmentation text, wherein the preset word segmentation model includes: a first word segmentation model, the first word segmentation model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: preset words, and part-of-speech tags of preset words, the part-of-speech tags include: a first part-of-speech tag indicating that the preset word belongs to the positive example data, and a second part-of-speech tag indicating that the preset word belongs to the breakpoint data, the first word segmentation model is used to truncate the text to be segmented at the position of the target word belonging to the breakpoint data; when the first number of the first word segmentation text is greater than the preset number threshold, the confidence of the first word segmentation model is obtained; when the confidence is greater than the preset confidence threshold, the first word segmentation text is determined as the target word segmentation text.

[0063] In the above embodiment of the present invention, the first word segmentation model is trained using preset words and corresponding part-of-speech tags. Therefore, in the process of segmenting the text to be segmented using the first word segmentation model, the first word segmentation model can be used to identify the part-of-speech tag corresponding to each target word in the text to be segmented, and then the target word whose part-of-speech tag belongs to the breakpoint data is determined in the text to be segmented, and the text to be segmented is staged at the position of the target word to obtain multiple first segmentation texts of the text to be segmented. Since the first word segmentation model uses machine learning to identify the part-of-speech tag of each target word, the recognition result of the part-of-speech tag may be incorrect, and the segmentation result based on the part-of-speech tag may also be incorrect. Then, the confidence of the first word segmentation model can be evaluated. If the confidence is greater than the preset confidence threshold, it means that the first segmentation text obtained by the first word segmentation model is credible, and then the first segmentation text is determined as the target segmentation text.

[0064] Optionally, the first word segmentation model can be a BERT model, such as a BERT-CRF (Conditional Random Field) model. The BERT-CRF model segments sentences with non-common components, and optimizes the traditional BIESO annotation format to IO format, where I represents positive data and O represents breakpoint data. This solution can greatly improve the generalization ability of compound sentence segmentation. For example, the text to be segmented is "check the weather, check the package, and check the balance", and the part-of-speech tags added are: "check I", "weather I", "air O", "check I", "package I", "meal O", "check I", "remaining I", and "amount O". It is truncated according to the position of the breakpoint data identified by the model, and is truncated into three first word segmentation texts, namely "check the weather", "check the package", and "check the balance".

[0065] As an optional embodiment, after using the first segmentation model to segment the text to be segmented to obtain the first segmentation text, the method also includes: using the second segmentation model to segment the text to be segmented according to semantics to obtain a second segmentation text, wherein the second segmentation model is used to determine the second segmentation text corresponding to each semantic when the text to be segmented includes multiple semantics, and each second segmentation text includes at least one target word; when the first number of the first segmentation texts is not greater than a preset number threshold, and the second number of the second segmentation texts is greater than the preset number threshold, use the second segmentation model to determine whether the second number of second segmentation texts has the same target semantics as the text to be segmented; when the target semantics are the same, determine the second segmentation text as the target segmentation text; when the target semantics are different, determine the first segmentation text as the target segmentation text.

[0066] In the above embodiment of the present invention, the second word segmentation model can be a word segmentation model that uses semantics for word segmentation. The second word segmentation model can be used to identify multiple semantics in the text to be segmented, and obtain a second word segmentation text for each semantic. However, the accuracy of the word segmentation result of the second word segmentation model is lower than that of the first word segmentation model. Therefore, when the confidence of the first word segmentation model is greater than the preset confidence threshold, the first word segmentation model is used for word segmentation in priority. When the confidence of the first word segmentation model is not greater than the preset confidence threshold, the second word segmentation model can be used to assist in word segmentation.

[0067] Optionally, the preset quantity threshold can be 1. If the first word segmentation model cannot segment the text to be segmented, the first word segmentation text output by the first word segmentation model is the text to be segmented. Therefore, the number of first word segmentation texts output by the first word segmentation model is 1. In this case, if the number of second word segmentation texts output by the second word segmentation model is greater than 1, it means that the second word segmentation model can be used to segment the text to be segmented. In order to verify the accuracy of the multiple second word segmentation texts output by the second word segmentation model, the second word segmentation model can be used to determine whether the semantics of all the second word segmentation texts output by the second word segmentation model are the same as those of the text to be segmented. If they are the same, it means that the second word segmentation text has passed the verification, and the second word segmentation text is determined as the target word segmentation text. If they are not the same, it means that the second word segmentation text has not passed the verification, and the first word segmentation text is determined as the target word segmentation text. When the first word segmentation text is the text to be segmented, the target word segmentation text is also the text to be segmented, indicating that the text to be segmented does not need to be segmented.

[0068] Optionally, the second word segmentation model can be a large language model LLM. With the emergence of the large language model LLM, the traditional recognition method of complex sentences can be integrated with the large model, which has good recognition results for complex queries with multiple intentions (such as text to be segmented).

[0069] As an optional embodiment, after using the second segmentation model to segment the text to be segmented according to semantics to obtain the second segmentation text, the method also includes: judging whether the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text; when the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, determining the second segmentation text as the target segmentation text; when the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, determining the preset segmentation text with the least quantity as the target segmentation text, wherein the preset segmentation text includes the first segmentation text and the second segmentation text.

[0070] In the above embodiment of the present invention, when the confidence of the first word segmentation model is not greater than the preset confidence threshold, the number of first word segmentation texts obtained by the first word segmentation model can be counted as the first number, and the number of second word segmentation texts obtained by the second word segmentation model can be counted as the second number. If the first number and the second number are the same, the second word segmentation text obtained by the second word segmentation model is preferentially used as the target word segmentation text. If the first number and the second number are the same, the word segmentation text with the smaller number is used. For example, if the first number is less than the second number, the first word segmentation text is used as the target word segmentation text; if the first number is greater than the second number, the second word segmentation text is used as the target word segmentation text.

[0071] The present invention also provides an optional embodiment, which provides a complex sentence understanding solution in an LLM-based intelligent customer service scenario. It utilizes a multi-path complex sentence understanding solution and combines it with a large model for complex sentence recognition. It can not only solve the problem of complex complex sentence understanding, but can also solve the problems of three-part and four-part sentences well. In addition, the migration cost is low, and it can be migrated to other businesses at low cost.

[0072] FIG2 is a schematic diagram of a complex sentence understanding process in an LLM-based intelligent customer service scenario according to an embodiment of the present invention. As shown in FIG2 , the process includes the following steps:

[0073] In step S21 , the user input text (such as the text to be segmented) is processed by a text preprocessing module to perform operations such as text error correction, uppercase and lowercase conversion, punctuation removal or normalization.

[0074] Step S22: using a complex sentence detection module to identify whether the text input by the user is a multi-intention complex sentence.

[0075] Step S23: If it is a single-sentence multi-intent text (i.e., a multi-intent compound sentence), multi-channel text segmentation recognition is performed to obtain a single-intent text. For example, the text to be segmented is distributed to a rule-based segmentation module for segmentation through the segmentation interface distribution module, and / or the text to be segmented is distributed to a BERT-based breakpoint recognition module for segmentation, and / or the text to be segmented is distributed to an LLM-based segmentation module for segmentation. If it is not a multi-intent compound sentence, it is a single-intent text, that is, the single-intent text is sent to the NLU semantic understanding module for semantic understanding.

[0076] Step S24: Finally, the three segmentation results are compared and integrated.

[0077] In step S24, multiple single-intent texts are sent to the NLU semantic understanding module in sequence for semantic understanding, and finally a command is issued.

[0078] As an optional embodiment, the text to be segmented is distributed to a rule-based segmentation module for segmentation, including a template common component segmentation scheme based on a tag dictionary tree, specifically including: a template (such as a preset vocabulary) construction process and a recognition process.

[0079] Optionally, the template construction process includes: constructing a preset vocabulary and a preset tag dictionary.

[0080] Optionally, construct the tag data needed for the template:

[0081] “Search_action:{search, check, find, search, query, ...}

[0082] Open_action: {open, open, ...}

[0083] Close_action: {Close, turn it on, close it, ...}

[0084] Conjunction: {and, and,...}

[0085] Modal_action:{help me, for me, give me, ...}

[0086] Stop_action:{Please, please, ...}

[0087] Entity: Introducing hanlp to identify noun vocabulary construction through part of speech".

[0088] Optionally, the tags are combined to form a batch of template formats (eg, to create a preset tag queue).

[0089] FIG3 is a schematic diagram of building a preset tag dictionary according to an embodiment of the present invention. As shown in FIG3 , a dictionary tree is generated for template data, and the result corresponds to the id field, which is expressed as:

[0090] "{"id":1,"result":[[k:Modal_action|][k:Search_action][e:Entity1]][[k:Modal_action|][k:Search_action][e: Entity2]],"template":[k:Modal_action|][k:Search_action][e:Entity1][k:Conjunction][e:Entity2],"rank":1}".

[0091] As an optional example, the recognition process includes: performing label recognition on the original text (such as the text to be segmented), traversing all preset word lists to identify the target label corresponding to each target word, such as "query: help me check the weather and traffic conditions", "help me: Modal_action", "check: search_action", "weather: entity", "and: Conjunction", "traffic conditions: entity".

[0092] For example, "help me," "check," "weather," and "traffic conditions" are converted into: "Modal_action," "search_action," "entity1," "Conjunction," and "entity2." This is matched against the preset tag dictionary. If a match is found, the recognition result corresponding to the tag template is extracted. The recognition result is extracted based on the template ID. If there are multiple results for that ID, they are sorted based on the rank field. The tag recognition results (also known as the target sub-tag queue) are: Target sub-tag queue 1. "Modal_action," "search_action," "search_actionentity1," Target sub-tag queue 1. "Modal_action," "search_action," "search_actionentity2." Finally, the above tag recognition results are converted into text: "Help me check the weather," "Help me check the traffic conditions," and recognition is complete.

[0093] Optionally, the text to be segmented is distributed to a BERT-based breakpoint recognition module for segmentation. The BERT-CRF model segments sentences with non-common components, and the traditional BIESO annotation format is introduced and optimized to the IO format, where I represents positive data and O represents breakpoint data. This solution can greatly improve the generalization ability of compound sentence segmentation. For example, the text to be segmented is "check the weather, check the package, and check the balance", and the added part-of-speech tags are: "check I", "weather I", "air O", "check I", "package I", "meal O", "check I", "remaining I", and "amount O". The data is truncated according to the position of the breakpoint data identified by the model, and is truncated into three first segmentation texts, namely "check the weather", "check the package", and "check the balance", and the process ends.

[0094] Optionally, the algorithm process for segmenting complex sentences based on the large model includes:

[0095] "Input: User input query and large model interface API

[0096] Output: Sentence segmentation results based on LLM recognition

[0097] The process is as follows:

[0098] Connect LLM through langchain to establish llm_connect / / Establish large model interface

[0099] For i=1:N / / Get user input data

[0100] Prompt(data(i)) / / Construct prompt sentence

[0101] initialize_agent(Prompt,llm_connect) / / Initialize the agent

[0102] Agent.Run() / / Start execution

[0103] Result() / / Get the segmentation result".

[0104] Alternatively, the Prompt construct example is:

[0105] “template="""

[0106] You are my tool assistant. I give you a sentence. If this sentence has multiple meanings, you help me split it into multiple texts that express the complete meaning. If it has only one meaning, there is no need to split it.

[0107] I'll give you a few cases

[0108] Input: Turn on the air conditioner and windows

[0109] Return: 1. Turn on the air conditioner

[0110] 2. Open the car windows

[0111] Input: Check weather, phone bill, and traffic

[0112] Return: 1. Check the weather

[0113] 2. Check phone bills

[0114] 3. Check traffic

[0115] Input: Today is a good day

[0116] Return: 1. Today is a good day

[0117] Now I am going to give you input: {query}

[0118] """

[0119] prompt=PromptTemplate.from_template(template)

[0120] prompt.format(src_lang="Check today's weather and traffic conditions")".

[0121] Optionally, the Agent construction includes:

[0122] "agent=initialize_agent(tools,llm,

[0123] agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,verbose=True)

[0124] agent.run("Check today's weather and traffic conditions")

[0125] Output

[0126] 1. Check today's weather

[0127] 2. Check traffic conditions

[0128] Adding LLM not only enhances the generalization ability of complex sentence segmentation, but also provides a certain text error correction capability of the large model. For example:

[0129] 1 Input: What's the weather like today? Check traffic conditions

[0130] 2. Output:

[0131] a) What is the weather like today?

[0132] b) Check traffic conditions."

[0133] Optionally, in step S24 above, the three complex sentence understanding solutions are integrated and can be directly applied to specific online businesses. The integration process includes:

[0134] Step S241: The priority of the segmentation result based on the tag dictionary tree is higher than that of all model results.

[0135] In step S242, the model segmentation results are compared and analyzed. If the BERT model recognizes a breakpoint, steps S243 to S245 are executed. If the BERT model recognizes no breakpoint, steps S246 and S247 are executed.

[0136] In step S243, if the BERT model confidence score is greater than 0.95, the BERT segmentation result is directly output.

[0137] Step S244: If the number of clauses in the BERT and LLM results is the same, the LLM recognition result is output.

[0138] Step S245: If the number of clauses in the BERT and LLM results is different, the strategy result with fewer clauses is output.

[0139] Step S246: If the number of LLM recognition clauses is greater than 1, it means that the LLM has results that need to be reconfirmed.

[0140] Step S247: The recognition result is input into the LLM judgment prompt again, and the design is as follows:

[0141] “template="""

[0142] You are my tool assistant. Help me identify whether these sentences and this long sentence express the same meaning. Just return yes or no.

[0143] enter:

[0144] 1. {Clause 1}, {Clause 2}, {Clause N}

[0145] 2.{original text}"""".

[0146] Step S248: The process ends and the segmentation result is output.

[0147] The technical solution provided in this application has been verified against three-party data sources and has achieved certain results. 500 compound sentences based on digital human / intelligent customer service online data were randomly selected as verification samples; 5000 test sets were manually constructed, and 2000 were sampled as verification samples; and 1000 open source data sets were randomly selected as verification samples.

[0148] Table 1 is a schematic table of accuracy comparison according to an embodiment of the present invention, and Table 2 is a schematic table of performance comparison according to an embodiment of the present invention. The comparison results of Table 1 and Table 2 show that the segmentation scheme based on the label dictionary tree (such as the preset label dictionary) is relatively limited in business, and has poor generalization for data not in the dictionary tree, but can be used to quickly repair online problems in online business, and has faster performance; the BERT-based compound sentence segmentation (such as the first word segmentation model) can basically meet the needs of online business, but cannot solve the segmentation in some complex error correction scenarios, and the overall effect is unsatisfactory in the voice scenario (ASR recognition error); the LLM-based compound sentence segmentation scheme (such as the second word segmentation model) has greater performance pressure, and due to the large number of LLM parameters, the user's perceived delay in the system is higher.

[0149] Table 1

[0150] Table 2

[0151] The technical solution proposed in this application integrates three strategies to achieve both performance and effectiveness. While sacrificing some performance, it achieves an overall accuracy rate of over 93%, while maintaining a significant decrease in user-perceived performance. Furthermore, it can accommodate a wider range of scenarios and incorporates a hotfix mechanism with a T+0 timeframe for online issues, improving user-friendliness for online businesses without impacting system performance.

[0152] The technical solution provided in this application provides a sentence segmentation method based on a tag dictionary tree, BERT, and LLM. The integration of the three methods not only solves the two methods of sentence segmentation: common component segmentation and non-common component segmentation, but also the introduction of BERT and LLM can greatly enhance the generalization ability of sentence segmentation. Compared with the traditional complex sentence understanding solution (multi-intent model solution), the invention of this article effectively solves the problem of semantic loss, can accurately understand and execute user instructions, and can effectively solve at least 90% of complex sentence understanding in intelligent customer service scenarios. In extreme cases, it can solve the problem of four-sentence segmentation of sentences, such as: checking traffic conditions, weather, traffic, and phone charges. It has a high promotion value and can be directly reused in scenarios such as intelligent voice, intelligent customer service, intelligent car dialogue, and smart speakers, and has a high application prospect.

[0153] Figure 4 is a schematic diagram of an overall customer service question-and-answer process according to an embodiment of the present invention. As shown in Figure 4, the user inputs the text to be segmented through the customer service front-end interaction module; the text to be segmented is transmitted to the text preprocessing module through the central control DCC module; the text processing module transmits the processed text to be segmented to the dialogue understanding module to determine whether the text to be segmented is a complex sentence. If so, it is first sent to the sentence segmentation module for segmentation, and then sent to the intention recognition module; otherwise, it is directly sent to the intention recognition module; then the intention recognition module performs intent recognition and the entity recognition module performs further recognition; then the reply text of the text to be segmented is obtained from the answer extraction module, and the reply text is returned to the customer service front-end interaction module.

[0154] The word segmentation method in the above embodiment can also be understood as a semantic segmentation method or a semantic analysis method.

[0155] According to an embodiment of the present invention, a word segmentation device embodiment is also provided. It should be noted that the word segmentation device can be used to execute the word segmentation method in the embodiment of the present invention, and the word segmentation method in the embodiment of the present invention can be executed in the word segmentation device.

[0156] FIG5 is a schematic diagram of a word segmentation device according to an embodiment of the present invention. As shown in FIG5 , the device may include: an acquisition module 52 for acquiring a text to be segmented, wherein the text to be segmented includes: a plurality of target words; an adding module 54 for adding a corresponding target tag for each target word according to a preset word list to obtain a target tag queue, wherein the preset word list is used to record a plurality of preset words and a preset tag corresponding to each preset word, the target tag is a preset tag corresponding to a preset word that matches the target word, and the target tag queue includes: a plurality of target tags arranged in a target order, and the target order is the order in which the target words corresponding to each target tag are arranged in the text to be segmented; a splitting module 54 for adding a corresponding target tag to each target word according to a preset word list to obtain a target tag queue, wherein the preset word list is used to record a plurality of preset words and a preset tag corresponding to each preset word, and the target tag queue includes: a plurality of target tags arranged in a target order, and the target order is the order in which the target words corresponding to each target tag are arranged in the text to be segmented; Block 56 is used to split the target tag queue into multiple target sub-tag queues according to the target word segmentation rules pre-configured in the preset tag dictionary, wherein the preset tag dictionary record is used to record multiple preset tag queues and the preset word segmentation rules corresponding to each preset tag queue, the target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue, and the preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that some preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues; the conversion module 58 is used to convert the target tag in each target sub-tag queue into a corresponding target word to obtain a target word segmentation text.

[0157] It should be noted that the acquisition module 52 in this embodiment can be used to execute step S102 in the embodiment of the present application, the addition module 54 in this embodiment can be used to execute step S104 in the embodiment of the present application, the splitting module 56 in this embodiment can be used to execute step S106 in the embodiment of the present application, and the conversion module 58 in this embodiment can be used to execute step S108 in the embodiment of the present application. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0158] In an embodiment of the present invention, a preset tag dictionary record is used to record the correspondence between a preset tag queue and a preset word segmentation rule. In the process of segmenting the text to be segmented using the preset tag dictionary, the correspondence between the preset words and the preset tags recorded in the preset word list can be used to add target tags to multiple target words in the text to be segmented, thereby obtaining a target tag queue. Then, a preset tag queue matching the target tag queue is queried in the preset tag dictionary, and the preset word segmentation rule corresponding to the preset tag queue is used as the target word segmentation rule of the target tag queue. Then, the target word segmentation rule is used to select some target tags from the target tag queue for arrangement, thereby obtaining multiple target sub-tag queues, and each target sub-tag queue may include target tags corresponding to some target words in the text to be segmented. By converting the target tags in the target sub-tag queue into corresponding target words, the target segmentation text of the text to be segmented can be obtained, thereby achieving the technical effect of accurate word segmentation of the text to be segmented, thereby solving the technical problem that the prior art cannot accurately segment complex sentences.

[0159] As an optional embodiment, the splitting module includes: a query submodule, which is used to query whether there is a preset tag queue matching the target tag queue in the preset tag dictionary; a first splitting submodule, which is used to use a preset word segmentation model to segment the text to be segmented to obtain a target word segmentation text when there is a preset tag queue matching the target tag queue in the preset tag dictionary; a second splitting submodule, which is used to determine the preset word segmentation rule corresponding to the preset tag queue matching the target tag queue as the target word segmentation rule when there is a preset tag queue matching the target tag queue in the preset tag dictionary, and split the target tag queue into multiple target sub-tag queues according to the target word segmentation rule.

[0160] As an optional embodiment, the query submodule includes: a first extraction unit, used to extract the target tag arranged in the first position from the target tag queue as the first target tag; a first judgment unit, used to traverse in the preset tag dictionary whether there is a first preset tag queue with the first position as the first target tag; a first placement unit, used to place the first preset tag queue into the first preset tag queue set if the first preset tag queue exists in the preset tag dictionary; and a first determination unit, used to determine that there is no preset tag queue in the preset tag dictionary that matches the target tag queue if the first preset tag queue does not exist in the preset tag dictionary.

[0161] As an optional embodiment, the device also includes: a second extraction unit, which is used to, if a first preset tag queue exists in the preset tag dictionary, put the first preset tag queue into the first preset tag queue set, and then extract the target tag arranged in the second position from the target tag queue as the second target tag, wherein the second position is located after the first position and adjacent to the first position; a second judgment unit, which is used to traverse the first preset tag queue set to see whether there is a second preset tag queue with the second position as the second target tag; a second placement unit, which is used to, if a second preset tag queue exists in the first preset tag queue set, put the second preset tag queue into the second preset tag queue set, and use the second position as the first position, and use the second preset tag queue set as the first preset tag queue set; a second determination unit, which is used to determine that there is no preset tag queue in the preset tag dictionary that matches the target tag queue, if the second preset tag queue does not exist in the first preset tag queue set.

[0162] As an optional embodiment, the device also includes: a detection unit, used to detect whether there is a target tag in the second position of the target tag queue before extracting the target tag arranged in the second position from the target tag queue as the second target tag; a third judgment unit, used to traverse whether there is a third preset tag queue in the first preset tag queue set that does not have a target tag in the second position when there is no target tag in the second position of the target tag queue; a third determination unit, used to determine the third preset tag queue as a preset tag queue that matches the target tag queue when there is a third preset tag queue in the first preset tag queue set; a fourth determination unit, used to determine that there is no preset tag queue in the preset tag dictionary that matches the target tag queue when there is no third preset tag queue in the first preset tag queue set.

[0163] As an optional embodiment, the first splitting submodule includes: a first word segmentation subunit, which is used to use a first word segmentation model to segment the text to be segmented to obtain a first word segmentation text, wherein the preset word segmentation model includes: a first word segmentation model, the first word segmentation model is trained by machine learning using multiple sets of data, and each set of data in the multiple sets of data includes: preset words, and part-of-speech tags of preset words, and the part-of-speech tags include: a first part-of-speech tag indicating that the preset word belongs to the positive example data, and a second part-of-speech tag indicating that the preset word belongs to the breakpoint data, and the first word segmentation model is used to truncate the text to be segmented at the position of the target word belonging to the breakpoint data; an acquisition subunit, which is used to obtain the confidence of the first word segmentation model when the first quantity of the first word segmentation text is greater than the preset quantity threshold; a first determination subunit, which is used to determine the first word segmentation text as the target word segmentation text when the confidence is greater than the preset confidence threshold.

[0164] As an optional embodiment, the device also includes: a second word segmentation subunit, which is used to segment the text to be segmented using the first word segmentation model to obtain the first word segmentation text, and then use the second word segmentation model to segment the text to be segmented according to semantics to obtain a second word segmentation text, wherein the second word segmentation model is used to determine the second word segmentation text corresponding to each semantic when the text to be segmented includes multiple semantics, and each second word segmentation text includes at least one target word; a first judgment subunit, which is used to use the second word segmentation model to determine whether the second number of second word segmentation texts is the same as the target semantics of the text to be segmented when the first number of the first word segmentation texts is not greater than a preset number threshold and the second number of the second word segmentation texts is greater than the preset number threshold; a second determination subunit, which is used to determine the second word segmentation text as the target word segmentation text when the target semantics are the same; and a third determination subunit, which is used to determine the first word segmentation text as the target word segmentation text when the target semantics are different.

[0165] As an optional embodiment, the device also includes: a second judgment subunit, which is used to segment the text to be segmented according to semantics using the second segmentation model to obtain the second segmentation text, and then determine whether the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text; a fourth determination subunit, which is used to determine the second segmentation text as the target segmentation text when the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text; and a fifth determination subunit, which is used to determine the preset segmentation text with the least quantity as the target segmentation text when the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, wherein the preset segmentation text includes the first segmentation text and the second segmentation text.

[0166] As an optional embodiment, the preset tag dictionary records the multiple preset tag queues in a tree form.

[0167] As an optional embodiment, the preset word segmentation model is a model trained through machine learning.

[0168] As an optional embodiment, the first word segmentation model is a BERT model.

[0169] As an optional embodiment, the second word segmentation model is a large language model LLM.

[0170] An embodiment of the present invention may provide an electronic device, such as a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0171] Optionally, in this embodiment, the computer terminal may be at least one network device among a plurality of network devices located in a computer network.

[0172] In this embodiment, the electronic device can execute the program code of the following steps in the word segmentation method: obtaining a text to be segmented, wherein the text to be segmented includes: multiple target words; adding a corresponding target tag to each target word according to a preset word table to obtain a target tag queue, wherein the preset word table is used to record multiple preset words and a preset tag corresponding to each preset word, and the target tag is a preset tag corresponding to a preset word matching the target word, and the target tag queue includes: multiple target tags arranged in a target order, and the target order is the arrangement order of the target words corresponding to each target tag in the text to be segmented; according to a target word segmentation rule pre-configured in a preset tag dictionary, splitting the target tag queue into multiple target sub-tag queues, wherein the preset tag dictionary records multiple preset tag queues and a preset word segmentation rule corresponding to each preset tag queue, and the target word segmentation rule is a preset word segmentation rule corresponding to the preset tag queue matching the target tag queue, and the preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that some preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues; converting the target tag in each target sub-tag queue into a corresponding target word to obtain a target word segmentation text.

[0173] Alternatively, Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present invention. As shown in Figure 6 , the electronic device 60 may include: one or more (only one is shown in the figure) processors 62 and a memory 64 .

[0174] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the word segmentation method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned word segmentation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal 60 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0175] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a text to be segmented, wherein the text to be segmented includes: multiple target words; adding a corresponding target tag to each target word according to a preset word table to obtain a target tag queue, wherein the preset word table is used to record multiple preset words and a preset tag corresponding to each preset word, and the target tag is a preset tag corresponding to a preset word matching the target word, and the target tag queue includes: multiple target tags arranged in a target order, and the target order is the order in which the target words corresponding to each target tag are arranged in the text to be segmented; according to a target segmentation rule pre-configured in a preset tag dictionary, splitting the target tag queue into multiple target sub-tag queues, wherein the preset tag dictionary records multiple preset tag queues and a preset segmentation rule corresponding to each preset tag queue, and the target segmentation rule is a preset segmentation rule corresponding to the preset tag queue matching the target tag queue, and the preset tag queue includes: multiple preset tags arranged in sequence, and the preset segmentation rule is used to indicate that some preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues; converting the target tag in each target sub-tag queue into a corresponding target word to obtain a target segmented text.

[0176] Optionally, the above-mentioned processor can also execute the program code of the following steps: query whether there is a preset tag queue matching the target tag queue in the preset tag dictionary; if there is no preset tag queue matching the target tag queue in the preset tag dictionary, use the preset word segmentation model to segment the text to be segmented to obtain the target word segmentation text; if there is a preset tag queue matching the target tag queue in the preset tag dictionary, determine the preset word segmentation rule corresponding to the preset tag queue matching the target tag queue as the target word segmentation rule, and split the target tag queue into multiple target sub-tag queues according to the target word segmentation rule.

[0177] Optionally, the processor may also execute the following program code: extracting the target tag arranged in the first position from the target tag queue as the first target tag; traversing the preset tag dictionary to see whether there is a first preset tag queue with the first target tag in the first position; if the first preset tag queue exists in the preset tag dictionary, placing the first preset tag queue into the first preset tag queue set; if the first preset tag queue does not exist in the preset tag dictionary, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0178] Optionally, the processor may also execute the following program code: extracting the target tag arranged in the second position from the target tag queue as the second target tag, wherein the second position is located after the first position and adjacent to the first position; traversing the first preset tag queue set to see whether there is a second preset tag queue with the second position as the second target tag; if there is a second preset tag queue in the first preset tag queue set, placing the second preset tag queue into the second preset tag queue set, and using the second position as the first position, and using the second preset tag queue set as the first preset tag queue set; if there is no second preset tag queue in the first preset tag queue set, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0179] Optionally, the processor may also execute the following program code: detecting whether there is a target tag at the second position of the target tag queue; if there is no target tag at the second position of the target tag queue, traversing the first preset tag queue set to see whether there is a third preset tag queue in which there is no target tag at the second position; if there is a third preset tag queue in the first preset tag queue set, determining the third preset tag queue as a preset tag queue that matches the target tag queue; if there is no third preset tag queue in the first preset tag queue set, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0180] Optionally, the processor may also execute the following program code: segmenting the text to be segmented using a first segmentation model to obtain a first segmented text, wherein the preset segmentation model includes: a first segmentation model, the first segmentation model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: preset words, and part-of-speech tags of the preset words, the part-of-speech tags include: a first part-of-speech tag indicating that the preset word belongs to the positive example data, and a second part-of-speech tag indicating that the preset word belongs to the breakpoint data, the first segmentation model is used to truncate the text to be segmented at the position of the target word belonging to the breakpoint data; when the first quantity of the first segmented text is greater than a preset quantity threshold, obtaining the confidence of the first segmentation model; when the confidence is greater than the preset confidence threshold, determining the first segmented text as the target segmented text.

[0181] Optionally, the processor may also execute the following program code: using a second word segmentation model to segment the text to be segmented according to semantics to obtain a second word segmentation text, wherein the second word segmentation model is used to determine the second word segmentation text corresponding to each semantic when the text to be segmented includes multiple semantics, and each second word segmentation text includes at least one target word; when the first number of the first word segmentation texts is not greater than a preset number threshold, and the second number of the second word segmentation texts is greater than the preset number threshold, using the second word segmentation model to determine whether the second number of second word segmentation texts has the same target semantics as the text to be segmented; when the target semantics are the same, determining the second word segmentation text as the target word segmentation text; when the target semantics are different, determining the first word segmentation text as the target word segmentation text.

[0182] Optionally, the processor may also execute the following program code: determining whether the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text; if the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, determining the second segmentation text as the target segmentation text; if the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, determining the preset segmentation text with the least quantity as the target segmentation text, wherein the preset segmentation text includes the first segmentation text and the second segmentation text.

[0183] Optionally, the preset tag dictionary records the multiple preset tag queues in a tree form.

[0184] Optionally, the preset word segmentation model is a model trained through machine learning.

[0185] Optionally, the first word segmentation model is a BERT model.

[0186] Optionally, the second word segmentation model is a large language model (LLM).

[0187] According to an embodiment of the present invention, a word segmentation scheme is provided. In the embodiment of the present invention, a preset tag dictionary is used to record the correspondence between a preset tag queue and a preset word segmentation rule. In the process of using the preset tag dictionary to segment the text to be segmented, the correspondence between the preset words and the preset tags recorded in the preset word list can be used to add target tags to multiple target words in the text to be segmented, thereby obtaining a target tag queue. Then, a preset tag queue matching the target tag queue is searched in the preset tag dictionary, and the preset word segmentation rule corresponding to the preset tag queue is used as the target word segmentation rule of the target tag queue. Then, the target word segmentation rule is used to select some target tags from the target tag queue for arrangement, thereby obtaining multiple target sub-tag queues, and each target sub-tag queue may include target tags corresponding to some target words in the text to be segmented. The target tags in the target sub-tag queue are converted into corresponding target words to obtain the target word segmentation text of the text to be segmented, thereby achieving the technical effect of accurate word segmentation of the text to be segmented, thereby solving the technical problem that the prior art cannot accurately segment complex sentences.

[0188] Those skilled in the art will appreciate that the structure shown in FIG6 is merely illustrative, and the electronic device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal device. FIG6 does not limit the structure of the above-mentioned electronic devices. For example, the electronic device 60 may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG6, or may have a configuration different from that shown in FIG6.

[0189] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile medium. The non-volatile storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0190] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the word segmentation method provided in the embodiment.

[0191] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0192] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: obtaining a text to be segmented, wherein the text to be segmented includes: a plurality of target words; adding a corresponding target tag to each target word according to a preset word list to obtain a target tag queue, wherein the preset word list is used to record a plurality of preset words and a preset tag corresponding to each preset word, the target tag is a preset tag corresponding to a preset word that matches the target word, and the target tag queue includes: a plurality of target tags arranged in a target order, the target order being the order in which the target words corresponding to each target tag are arranged in the text to be segmented; according to The target word segmentation rules pre-configured in the preset tag dictionary split the target tag queue into multiple target sub-tag queues, wherein the preset tag dictionary record is used to record multiple preset tag queues and the preset word segmentation rules corresponding to each preset tag queue. The target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue. The preset tag queue includes: multiple preset tags arranged in sequence. The preset word segmentation rule is used to indicate that some preset tags in the preset tag queue are selected and arranged to obtain multiple preset sub-tag queues; the target tag in each target sub-tag queue is converted into a corresponding target word to obtain a target word segmentation text.

[0193] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: querying whether there is a preset tag queue matching the target tag queue in the preset tag dictionary; if there is no preset tag queue matching the target tag queue in the preset tag dictionary, using a preset word segmentation model to segment the text to be segmented to obtain a target word segmentation text; if there is a preset tag queue matching the target tag queue in the preset tag dictionary, determining the preset word segmentation rule corresponding to the preset tag queue matching the target tag queue as the target word segmentation rule, and splitting the target tag queue into multiple target sub-tag queues according to the target word segmentation rule.

[0194] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: extracting the target tag arranged in the first position from the target tag queue as the first target tag; traversing the preset tag dictionary to see whether there is a first preset tag queue with the first position as the first target tag; if the first preset tag queue exists in the preset tag dictionary, placing the first preset tag queue into a first preset tag queue set; if the first preset tag queue does not exist in the preset tag dictionary, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0195] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: extracting the target tag arranged in the second position from the target tag queue as the second target tag, wherein the second position is located after the first position and adjacent to the first position; traversing the first preset tag queue set to see whether there is a second preset tag queue with the second position as the second target tag; if there is a second preset tag queue in the first preset tag queue set, putting the second preset tag queue into the second preset tag queue set, and using the second position as the first position, and using the second preset tag queue set as the first preset tag queue set; if there is no second preset tag queue in the first preset tag queue set, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0196] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: detecting whether there is a target tag at the second position of the target tag queue; if there is no target tag at the second position of the target tag queue, traversing the first preset tag queue set to see whether there is a third preset tag queue in which there is no target tag at the second position; if there is a third preset tag queue in the first preset tag queue set, determining the third preset tag queue as a preset tag queue that matches the target tag queue; if there is no third preset tag queue in the first preset tag queue set, determining that there is no preset tag queue in the preset tag dictionary that matches the target tag queue.

[0197] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: using a first word segmentation model to segment the text to be segmented to obtain a first word segmentation text, wherein the preset word segmentation model includes: a first word segmentation model, the first word segmentation model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: preset words, and part-of-speech tags of the preset words, the part-of-speech tags include: a first part-of-speech tag indicating that the preset word belongs to the positive example data, and a second part-of-speech tag indicating that the preset word belongs to the breakpoint data, the first word segmentation model is used to truncate the text to be segmented at the position of the target word belonging to the breakpoint data; when the first number of the first word segmentation text is greater than the preset number threshold, obtain the confidence of the first word segmentation model; when the confidence is greater than the preset confidence threshold, determine the first word segmentation text as the target word segmentation text.

[0198] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: using a second word segmentation model to segment the text to be segmented according to semantics to obtain a second word segmentation text, wherein the second word segmentation model is used to determine the second word segmentation text corresponding to each semantic when the text to be segmented includes multiple semantics, and each second word segmentation text includes at least one target word; when the first number of the first word segmentation texts is not greater than a preset number threshold and the second number of the second word segmentation texts is greater than the preset number threshold, use the second word segmentation model to determine whether the second number of second word segmentation texts has the same target semantics as the text to be segmented; when the target semantics are the same, determine the second word segmentation text as the target word segmentation text; when the target semantics are different, determine the first word segmentation text as the target word segmentation text.

[0199] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining whether the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text; when the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, determining the second segmentation text as the target segmentation text; when the first quantity of the first segmentation text is the same as the second quantity of the second segmentation text, determining the preset segmentation text with the least quantity as the target segmentation text, wherein the preset segmentation text includes the first segmentation text and the second segmentation text.

[0200] Optionally, the preset tag dictionary records the multiple preset tag queues in a tree form.

[0201] Optionally, the preset word segmentation model is a model trained through machine learning.

[0202] Optionally, the first word segmentation model is a BERT model.

[0203] Optionally, the second word segmentation model is a large language model (LLM).

[0204] An embodiment of the present invention also provides a computer program product.

[0205] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the word segmentation method according to the above embodiment:

[0206] Acquire a text to be segmented, wherein the text to be segmented includes: a plurality of target words;

[0207] According to a preset vocabulary, a corresponding target tag is added to each target word to obtain a target tag queue, wherein the preset vocabulary is used to record multiple preset words and a preset tag corresponding to each preset word, the target tag is a preset tag corresponding to the preset word matching the target word, and the target tag queue includes: multiple target tags arranged in a target order, and the target order is the arrangement order of the target words corresponding to each target tag in the text to be segmented;

[0208] According to the target word segmentation rule pre-configured in the preset tag dictionary, the target tag queue is split into multiple target sub-tag queues, wherein the preset tag dictionary record is used to record multiple preset tag queues and the preset word segmentation rule corresponding to each preset tag queue, the target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue, the preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that a portion of the preset tags in the preset tag queue is selected and arranged to obtain multiple preset sub-tag queues;

[0209] The target tag in each target sub-tag queue is converted into the corresponding target word to obtain the target word segmentation text.

[0210] The features described in the embodiment of the aforementioned word segmentation method are applicable to the embodiment of the computer program product. The preferred implementation of the embodiment of the computer program product can refer to the relevant description in the embodiment of the aforementioned word segmentation method, which will not be repeated here.

[0211] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0212] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0214] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0215] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0216] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and other media that can store program code.

[0217] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

[0218] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0219] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A word segmentation method, comprising: Acquire a text to be segmented, wherein the text to be segmented includes: a plurality of target words; According to a preset vocabulary, a corresponding target tag is added to each target word to obtain a target tag queue, wherein the preset vocabulary is used to record multiple preset words and a preset tag corresponding to each preset word, the target tag is a preset tag corresponding to the preset word matching the target word, and the target tag queue includes: multiple target tags arranged in a target order, and the target order is the arrangement order of the target words corresponding to each target tag in the text to be segmented; According to the target word segmentation rule pre-configured in the preset tag dictionary, the target tag queue is split into multiple target sub-tag queues, wherein the preset tag dictionary record is used to record multiple preset tag queues and the preset word segmentation rule corresponding to each preset tag queue, the target word segmentation rule is the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue, the preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that a portion of the preset tags in the preset tag queue is selected and arranged to obtain multiple preset sub-tag queues; The target tag in each target sub-tag queue is converted into the corresponding target word to obtain the target word segmentation text.

2. The method according to claim 1, wherein According to the target word segmentation rules pre-configured in the preset tag dictionary, splitting the target tag queue into multiple target sub-tag queues includes: Querying the preset tag dictionary to determine whether there is a preset tag queue that matches the target tag queue; If there is no preset tag queue matching the target tag queue in the preset tag dictionary, segment the text to be segmented using a preset word segmentation model to obtain a target segmented text; In the case that there is a preset tag queue that matches the target tag queue in the preset tag dictionary, the preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue is determined as the target word segmentation rule, and the target tag queue is split into multiple target sub-tag queues according to the target word segmentation rule.

3. The method according to claim 2, wherein: Querying whether there is a preset tag queue matching the target tag queue in the preset tag dictionary includes: Extract the target tag arranged at the first position from the target tag queue as the first target tag; Traversing the preset tag dictionary to determine whether there is a first preset tag queue in which the first position is the first target tag; If the first preset tag queue exists in the preset tag dictionary, put the first preset tag queue into a first preset tag queue set; In a case where the first preset tag queue does not exist in the preset tag dictionary, it is determined that the preset tag queue matching the target tag queue does not exist in the preset tag dictionary.

4. The method according to claim 3, wherein: In a case where the first preset tag queue exists in the preset tag dictionary, after placing the first preset tag queue into a first preset tag queue set, the method further includes: Extracting the target tag arranged at a second position from the target tag queue as a second target tag, wherein the second position is located after the first position and adjacent to the first position; Traversing the first preset tag queue set to determine whether there is a second preset tag queue in which the second position is the second target tag; If the second preset tag queue exists in the first preset tag queue set, the second preset tag queue is placed in the second preset tag queue set, and the second position is used as the first position, and the second preset tag queue set is used as the first preset tag queue set; In a case where the second preset tag queue does not exist in the first preset tag queue set, it is determined that the preset tag queue matching the target tag queue does not exist in the preset tag dictionary.

5. The method according to claim 4, wherein Before extracting the target tag arranged at the second position from the target tag queue as the second target tag, the method further includes: Detecting whether the target tag exists at the second position of the target tag queue; When the target tag does not exist at the second position of the target tag queue, searching the first preset tag queue set to see whether there is a third preset tag queue in which the target tag does not exist at the second position; In a case where the third preset tag queue exists in the first preset tag queue set, determining the third preset tag queue as the preset tag queue matching the target tag queue; In a case where the third preset tag queue does not exist in the first preset tag queue set, it is determined that the preset tag queue matching the target tag queue does not exist in the preset tag dictionary.

6. The method according to claim 2, wherein: When there is no preset tag queue matching the target tag queue in the preset tag dictionary, segmenting the text to be segmented using a preset segmentation model to obtain a target segmented text includes: Segmenting the text to be segmented using a first segmentation model to obtain a first segmented text, wherein the preset segmentation model includes: the first segmentation model, the first segmentation model is trained by machine learning using multiple sets of data, each set of data in the multiple sets of data includes: a preset word, and a part-of-speech tag of the preset word, the part-of-speech tag includes: a first part-of-speech tag indicating that the preset word belongs to positive example data, and a second part-of-speech tag indicating that the preset word belongs to breakpoint data, and the first segmentation model is used to truncate the text to be segmented at the position of the target word belonging to the breakpoint data; When the first number of the first segmented texts is greater than a preset number threshold, obtaining the confidence of the first segmented model; When the confidence level is greater than a preset confidence threshold, the first segmented text is determined as the target segmented text.

7. The method according to claim 6, wherein: After segmenting the text to be segmented using the first segmentation model to obtain a first segmented text, the method further includes: Segmenting the text to be segmented according to semantics using a second segmentation model to obtain second segmented texts, wherein the second segmentation model is used to determine, when the text to be segmented includes multiple semantics, a second segmented text corresponding to each semantic, each second segmented text including at least one target word; When a first number of the first segmented texts is not greater than the preset number threshold, and a second number of the second segmented texts is greater than the preset number threshold, using the second segmented model to determine whether the second number of the second segmented texts has the same target semantics as the text to be segmented; In the case where the target semantics are the same, determining the second segmented text as the target segmented text; In the case where the target semantics are different, the first segmented text is determined as the target segmented text.

8. The method according to claim 7, wherein: After segmenting the text to be segmented according to semantics using the second segmentation model to obtain a second segmented text, the method further includes: Determining whether a first quantity of the first segmented text is the same as a second quantity of the second segmented text; When the first number of the first segmented texts is the same as the second number of the second segmented texts, determining the second segmented texts as the target segmented texts; When the first number of the first segmented texts is the same as the second number of the second segmented texts, the preset segmented texts with the least number are determined as the target segmented texts, wherein the preset segmented texts include the first segmented text and the second segmented text.

9. The method according to claim 1, wherein The preset tag dictionary records the plurality of preset tag queues in a tree form.

10. The method according to claim 1, wherein The preset word segmentation model is a model trained through machine learning.

11. The method according to claim 6, wherein: The first word segmentation model is a BERT model.

12. The method according to claim 7, wherein: The second word segmentation model is a large language model LLM.

13. A word segmentation device, comprising: An acquisition module is used to acquire a text to be segmented, wherein the text to be segmented includes: a plurality of target words; An adding module is used to add a corresponding target tag for each target word according to a preset vocabulary to obtain a target tag queue, wherein the preset vocabulary is used to record multiple preset words and a preset tag corresponding to each preset word, the target tag is a preset tag corresponding to the preset word matching the target word, and the target tag queue includes: multiple target tags arranged in a target order, and the target order is the arrangement order of the target words corresponding to each target tag in the text to be segmented; A splitting module is configured to split the target tag queue into multiple target sub-tag queues according to a target word segmentation rule pre-configured in a preset tag dictionary, wherein the preset tag dictionary is used to record multiple preset tag queues and a preset word segmentation rule corresponding to each preset tag queue, the target word segmentation rule is a preset word segmentation rule corresponding to the preset tag queue that matches the target tag queue, the preset tag queue includes: multiple preset tags arranged in sequence, and the preset word segmentation rule is used to indicate that a portion of the preset tags in the preset tag queue is selected and arranged to obtain multiple preset sub-tag queues; The conversion module is used to convert the target tag in each target sub-tag queue into the corresponding target word to obtain a target word segmentation text.

14. A non-volatile storage medium, wherein: The non-volatile storage medium is used to store a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the word segmentation method according to any one of claims 1 to 8.

15. An electronic device comprising: A memory and a processor, wherein the processor is used to run a program stored in the processor, wherein the word segmentation method according to any one of claims 1 to 8 is executed when the program is run.

16. A computer program product comprising a computer program, wherein When the computer program is executed by a processor, the steps of the word segmentation method according to any one of claims 1 to 12 are implemented.

Images