Text segmentation method and device, computer program product, and electronic device

Abstract

The present disclosure relates to the field of information processing, and provides a text segmentation method and device, a computer program product, and an electronic device. The method includes: segmenting a text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; based on the similarity between adjacent first target text segments in the first target text segment set, grouping the first target text segments in the first target text segment set to obtain a text segment grouping result; and determining the segmentation result of the text to be processed according to the text segment grouping result. The present disclosure can segment long texts according to semantics through the similarity between adjacent text segments, thereby improving the accuracy of text segmentation.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to a text segmentation method, a text segmentation device, a computer program product, and an electronic device. Background Art

[0002] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. Natural language processing relies on language models, and the main corpus data of language models is text. Usually, the size of the text window of the hyperparameters trained during language model training is limited, such as 4K tokens (word segmentation). Therefore, long texts usually need to be segmented to meet training requirements.

[0003] Text segmentation methods in the related art include segmentation based on length, segmentation based on symbols, segmentation based on a combination of length and symbols, and segmentation based on special formats.

[0004] However, the segmentation method based on length will cause the problem of semantic incoherence of the segmentation points; the segmentation based on symbols is difficult to ensure that the length after segmentation meets the training requirements; the segmentation method based on the combination of length and symbols has the problem of large semantic differences with all the segments. For example, a certain text segment may contain both relevant information such as the directory and the main text information of the first chapter, and various symbol characteristics need to be found during segmentation, which has poor versatility and high cost; segmentation based on special formats is only suitable for documents in special formats, so it also has the problem of poor versatility.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a text segmentation method and device, a computer program product and an electronic device, thereby improving the accuracy of text segmentation at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a text segmentation method is provided, comprising: segmenting a text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; grouping the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments in the first target text segment set to obtain a text segment grouping result; and determining a segmentation result of the text to be processed according to the text segment grouping result.

[0009] Optionally, in the case where the universal symbols include the punctuation marks, segmenting the text to be processed according to the universal symbols to obtain a first target text segment set includes: segmenting the text to be processed according to the punctuation marks in the text to be processed to obtain a first target text segment set.

[0010] Optionally, in the case where the universal symbol includes the line break symbol, segmenting the text to be processed according to the universal symbol to obtain a first target text segment set includes: segmenting the text to be processed according to the number of consecutive line break symbols in the text to be processed to obtain a first target text segment set.

[0011] Optionally, segmenting the text to be processed according to the number of consecutive line break symbols in the text to be processed to obtain a first target text segment set includes: determining the number of consecutive common symbols in the text to be processed, sorting the number of symbols in descending order to obtain a symbol number set; traversing the number of symbols in the symbol number set, and segmenting the text to be processed according to the number of consecutive line break symbols traversed to obtain the first target text segment set.

[0012] Optionally, traversing the number of symbols in the symbol quantity set and segmenting the text to be processed according to the number of consecutive line break symbols traversed to obtain the first target text segment set includes: determining the segmented text segment set based on the segmented text segments obtained by segmenting the text to be processed according to the number of consecutive line break symbols traversed last time; traversing the segmented text segments in the segmented text segment set based on the number of symbols currently traversed, and when the text length of the traversed segmented text segment is greater than the first preset text length, segmenting the segmented text segment according to the number of consecutive line break symbols traversed to update the segmented text segment set; obtaining the first target text segment set based on the segmented text segment set updated according to the number of consecutive line break symbols traversed last time.

[0013] Optionally, the grouping processing of the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments to obtain a text segment grouping result includes: grouping the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments and a similarity threshold to obtain a grouping processing result; successively updating the similarity threshold in a decreasing manner according to the similarity threshold, and for each group in the previous grouping processing result, grouping the first target text segments in the group based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update to update the grouping processing result until the prediction condition is met; and obtaining the text segment grouping result based on the last updated grouping processing result.

[0014] Optionally, the grouping processing of the first target text segments based on the similarity between adjacent first target text segments in the first target text segment set and the similarity threshold after the current update to update the grouping processing result includes: for each group in the previous grouping processing result, when the text length of the first target text segments in the group is greater than the second preset text length, grouping processing is performed on the first target text segments in the group based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update; otherwise, the first target text segments in the group are not grouped to update the grouping processing result.

[0015] Optionally, when the text length of the first target text segments in the group is greater than the second preset text length, the first target text segments in the group are grouped based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update, including: when the similarity between adjacent first target text segments in the group is less than or equal to the similarity threshold after the current update, determining that the adjacent first target text segments in the group belong to the same group; otherwise, determining that the adjacent first target text segments in the group belong to different groups.

[0016] Optionally, the text segment grouping result obtained based on the grouping processing result updated last time includes: for each group in the grouping processing result updated last time, splicing the first target text segments in the group to obtain the second target text segments; for each second target text segment, obtaining the feature value corresponding to the text segment quality feature in the second target text segment, inputting the feature value into the text quality prediction model, and determining the text quality score of the second target text segment; filtering out the second target text segments whose text quality scores are less than the text quality threshold, and obtaining the text segment grouping result based on the filtered second target text segments; wherein the text segment quality features include one or more of preset keywords, text segment length, proportion of punctuation marks in the text segment, proportion of pause words in the text segment, and text perplexity.

[0017] Optionally, determining the segmentation result of the text to be processed based on the text segment grouping result includes: when the total length of the text segments in K consecutive groups is less than or equal to a second preset text length, merging the K consecutive groups to obtain a merged text segment grouping result; determining the segmentation result of the text to be processed based on each grouped text segment in the merged text segment grouping result; wherein the K consecutive groupings minimize the sum of the differences between the total length of the text segments in each group in the merged text segment grouping result and the second preset text length.

[0018] According to a second aspect of the present disclosure, a text segmentation device is provided, comprising: a symbol segmentation module, configured to segment a text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; a similarity grouping module, configured to group the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments in the first target text segment set to obtain a text segment grouping result; and a segmentation result determination module, configured to determine a segmentation result of the text to be processed according to the text segment grouping result.

[0019] According to a third aspect of the present disclosure, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the text segmentation method as described in the first aspect.

[0020] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text segmentation method as described in the first aspect of the above embodiment.

[0021] It can be seen from the above technical solutions that the text segmentation method, text segmentation device, and computer program product and electronic device implementing the text segmentation method in the exemplary embodiment of the present disclosure have at least the following advantages and positive effects:

[0022] In the technical solutions provided by some embodiments of the present disclosure, on the one hand, after the text is segmented based on common symbols, the text segments are grouped according to the similarity of the text segments to ensure that texts with the same semantics are segmented into the same text segment, and texts with different semantics are segmented into different text segments, thereby improving the semantic accuracy of text segmentation; on the other hand, common symbols and text segment similarity are applicable to most texts, thus ensuring the universality of the text segmentation method.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0025] Figure 1 A schematic diagram showing an exemplary system architecture to which the text segmentation method and text segmentation device in the embodiments of the present disclosure can be applied;

[0026] Figure 2 A flowchart showing a text segmentation method in an exemplary embodiment of the present disclosure is shown;

[0027] Figure 3 A flow chart showing a method for obtaining a first target text segment set according to the number of consecutive line break symbols in an exemplary embodiment of the present disclosure is shown;

[0028] Figure 4 A flowchart diagram showing another method for obtaining a first target text segment set in an exemplary embodiment of the present disclosure;

[0029] Figure 5 A schematic flow chart showing a method for obtaining text segment grouping results in an exemplary embodiment of the present disclosure;

[0030] Figure 6 A flow chart showing a method for cleaning a grouping processing result to obtain a text segment grouping result in an exemplary embodiment of the present disclosure;

[0031] Figure 7 A schematic flow chart showing a method for bucketing a text segment in an exemplary embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram showing a flow chart of another text segmentation method in an exemplary embodiment of the present disclosure;

[0033] Fig. 9 A schematic diagram showing the composition of a text segmentation device in an exemplary embodiment of the present disclosure;

[0034] Fig.10 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0036] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0037] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0038] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. Natural language processing relies on language models, which are usually large-scale neural network models. Large-scale neural network models have a huge number of parameters and can capture, learn and simulate complex data patterns and human behaviors.

[0039] The main corpus data of language models are texts, which vary in length. Long language texts such as books may contain millions of Chinese characters, while short language texts such as questions and answers may only contain dozens of characters. Usually, the hyperparameter window size of language models during training is limited, such as 4K tokens. Therefore, long texts that exceed the window size usually need to be segmented to meet training requirements.

[0040] A piece of text or a text file has several components, including sentences that can be further divided into clauses, phrases, and words. Text segmentation can be understood as the process of breaking down or splitting text data into smaller and meaningful components, such as the process of segmenting a long text into multiple text segments.

[0041] Text segmentation methods in the related art include segmentation based on length, segmentation based on symbols, segmentation based on a combination of length and symbols, and segmentation based on special formats.

[0042] Length-based segmentation can be understood as segmenting the text according to a fixed length. For example, if the segmentation length is set to 512 characters, the text will be segmented into segments of 512 characters. This segmentation method usually causes context incoherence at the segmentation point and semantic loss. For example, for a text "A 63-year-old woman with a 20-year history of hemodialysis (HD) had a severe painful skin ulcer on her right lower leg..." segmentation based on length may segment the "pain" in "because of a severe painful skin ulcer on the right lower leg" into different segments before and after, resulting in semantic incoherence.

[0043] Symbol-based segmentation is segmentation based on symbols, such as segmentation based on periods, semicolons, etc. Although it may retain certain semantics, the length of the segmentation may not meet the requirements. For example, segmentation based on periods may result in very short paragraphs or very long paragraphs. When training a language model, text that is too long or too short will affect the decrease of the model's loss function, thereby affecting the performance of the language model.

[0044] Although the segmentation method based on length and symbol can retain certain semantics and the length of the segment may meet the requirements, there will be a problem of large semantic differences between all segments. For example, if a book is segmented according to line breaks and the length is required to be 10,000 characters, the first segment may contain relevant information such as the directory and the main text of the first chapter. In addition, due to the variety of texts, segmentation according to symbols requires finding various symbol characteristics, which has poor versatility and is relatively costly.

[0045] Special format-based segmentation can only segment text with special formats, such as markdown (a lightweight markup language), html (Hyper Text Markup Language), etc., so it has poor versatility. At the same time, it also has the problem of segmentation length, and the segmented text is sometimes too long or too short.

[0046] In order to solve the above problems, the present invention provides a text segmentation method and device, which can be applied to Figure 1 In the system architecture of the exemplary application environment shown.

[0047] like Figure 1 As shown, the system architecture 100 may include a terminal device 110 and a server 120. The terminal device 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, a smart wearable device, etc. The server 120 generally refers to a background system that provides services related to the text segmentation method in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal device 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0048] In an exemplary embodiment, the above-mentioned text segmentation method can be performed by the terminal device 110. Accordingly, the text segmentation device can be set in the terminal device 110 to implement the corresponding module functions. For example, the user determines the text to be processed in the terminal device 110, and the terminal device 110 segments the text to be processed according to the common symbols to obtain a first target text segment set, and based on the similarity between adjacent first target text segments in the first target text segment set, the first target text segments in the first target text segment set are grouped to obtain a text segment grouping result, and then the segmentation result of the text to be processed is determined according to the text segment grouping result.

[0049] In an exemplary embodiment, the above-mentioned text segmentation method can be executed by the server 120. Accordingly, the text segmentation device can be set in the server 120 to realize the corresponding module functions. For example, after the user selects the text to be processed in the terminal device 110 or the user selects the text to be processed in the terminal device 110, the terminal device 110 uploads the text to be processed to the server 120, and the server 120 segments the text to be processed according to the above-mentioned text segmentation method, and returns the segmentation result of the text to be processed to the terminal device 110.

[0050] In an exemplary embodiment, the above-mentioned text segmentation method can also be performed jointly by the terminal device 110 and the server 120. Accordingly, some modules of the text segmentation device can be set in the terminal device 110, and some modules can be set in the server 120 to realize the corresponding module functions. For example, after the user selects the text to be processed in the terminal device 110, the terminal device 110 first segments the text to be processed according to the universal symbols to obtain a first target text segment set, and then the terminal device 110 uploads the first target text segment set to the server 120, and the server 120 calculates the similarity between adjacent first target text segments in the first target text segment set, and then based on the similarity between adjacent first target text segments in the first target text segment set, the first target text segments in the first target text segment set are grouped to obtain a text segment grouping result, and according to the text segment grouping result, the segmentation result of the text to be processed is determined, and the segmentation result is returned to the terminal device 110.

[0051] It should be understood that Figure 1 The number of terminal devices and servers in the example is only illustrative. Any number of terminal devices and servers may be provided as required. For example, server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0052] Figure 2 A flowchart of a text segmentation method in an exemplary embodiment of the present disclosure is shown. Figure 2The method includes: step S210, segmenting the text to be processed according to common symbols to obtain a first target text segment set; step S220, grouping the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments in the first target text segment set to obtain a text segment grouping result; step S230, determining the segmentation result of the text to be processed according to the text segment grouping result; wherein the common symbols include one or more of punctuation marks and line break marks.

[0053] exist Figure 2 In the technical solution provided by the illustrated embodiment, on the one hand, after the text is segmented based on common symbols, the text segments are grouped according to the similarity of the text segments to ensure that texts with the same semantics are segmented into the same text segment, and texts with different semantics are segmented into different text segments, thereby improving the semantic accuracy of text segmentation; on the other hand, common symbols and text segment similarity are applicable to most texts, thus ensuring the universality of the text segmentation method.

[0054] The following Figure 2 The specific implementation methods of each step in the embodiment shown are described in detail:

[0055] In step S210, the text to be processed is segmented according to the common symbols to obtain a first target text segment set.

[0056] In an exemplary embodiment, the common symbols include one or more of punctuation marks and line break marks, wherein the punctuation marks may include commas, periods, exclamation marks, and the like.

[0057] In an exemplary embodiment, the text to be processed in the present disclosure may include long texts for large language model training, and may also include other texts that need to be segmented, which is not particularly limited in this exemplary embodiment.

[0058] A large model can be understood as a neural network model that contains ultra-large-scale parameters (usually more than one billion). It has a huge scale, can perform multi-task learning, and has powerful computing resources. It also requires a large amount of data for training. Only a large amount of data can give full play to the parameter scale advantage of the model.

[0059] Large models are widely used in the field of natural language processing, and are completely changing the state of natural language processing tasks, giving birth to more powerful and intelligent language technologies. Large models are one of the important directions for the development of artificial intelligence. At the same time, large models also have the ability to perform well in various natural language processing tasks, such as text classification, sentiment analysis, summary generation, translation, etc. Large models can be used in multiple application areas such as automatic writing, chatbots, virtual assistants, voice assistants, automatic translation, etc. The large model used in natural language processing can be called a large language model. As mentioned earlier, the training hyperparameter window size of the large language training model is limited, so for texts exceeding the window size, segmentation processing is required to meet the data requirements of the large language training model.

[0060] Exemplarily, in the case where the universal symbols include the punctuation marks, segmenting the text to be processed according to the universal symbols to obtain a first target text segment set includes: segmenting the text to be processed according to the punctuation marks in the text to be processed to obtain a first target text segment set.

[0061] Taking the punctuation mark as a period as an example, the text to be processed can be traversed, and when the period is traversed, the text before and after the period is segmented according to the traversed period, so that the text to be processed is divided into M sentences based on the period, and each sentence is a text segment, so as to obtain the first target text segment set. That is, there are M text segments in the first target text segment set.

[0062] Exemplarily, in the case where the universal symbols include the line break symbols, segmenting the text to be processed according to the universal symbols to obtain the first target text segment set includes: segmenting the text to be processed according to the number of consecutive line break symbols in the text to be processed to obtain the first target text segment set.

[0063] For example, books, papers, news and other texts usually exist in the form of line breaks, that is, different parts are usually separated by line break symbols. For example, long texts can usually be divided into chapters, etc., and there may be one or more line breaks between chapters, and sections may also be separated by one or more line breaks. Therefore, the text to be processed can be segmented by line break symbols to ensure that the content of the same chapter will not be divided into different text segments, thereby ensuring the coherence of the text segments and improving the accuracy of text segmentation.

[0064] For example, Figure 3 A flowchart of a method for obtaining a first target text segment set according to the number of consecutive line break symbols in an exemplary embodiment of the present disclosure is shown. Figure 3 The method may refer to step S310 to step S320. Wherein:

[0065] In step S310, the number of consecutive line break symbols in the text to be processed is determined, and the number of symbols is sorted in descending order to obtain a set of symbol numbers.

[0066] As mentioned above, text can usually be divided into chapters, etc., and chapters or sections may be separated by one or more line breaks. Usually, the same chapter represents a complete semantic segment, so segmentation by line breaks can avoid segmenting the content of the same chapter into different text segments, thereby improving the semantic accuracy of text segmentation.

[0067] Usually there may be spaces between line break symbols. In order to ensure the accuracy of segmentation, before segmenting the text to be processed, the spaces between consecutive line break symbols can be cleaned, that is, the spaces between consecutive line break symbols can be deleted. Then, the consecutive line break symbols are extracted from the cleaned text to be processed to count the number of consecutive line break symbols.

[0068] For example, the text to be processed can be traversed to count the number of continuous line break symbols. That is, if the previous traversed content is a line break symbol, and the next traversed content is also a line break symbol, then they are continuous line break symbols. If the content between two line break symbols is a space, all spaces between the two line break symbols can be deleted, so that the two line break symbols are also continuous line break symbols. In other words, if the traversed content between two line break symbols is only a space, then the two line break symbols are also continuous. In addition, 1 line break is also considered a continuous line break symbol.

[0069] For example, for a certain text, the number of consecutive line break symbols traversed in sequence is 1, 2, 3, 5, 3, 2, 2, 6, 8, 4, 5, 6. The number of line break symbols traversed is first deduplicated, and then the number of line break symbols after deduplication is sorted in descending order from large to small, and the resulting symbol number set is {8, 6, 5, 4, 3, 2, 1}. The symbol number set can be a list, which can be represented by line_feed_num_list, such as line_feed_num_list = [8, 6, 5, 4, 3, 2, 1].

[0070] When extracting the number of consecutive line break symbols in the text to be processed, the extracted number of symbols can also be stored in list A in sequence, and then after the extraction is completed, list A is deduplicated, and after the deduplication is completed, the elements in list A are sorted in descending order to obtain list B. List B can be understood as the above-mentioned symbol number set.

[0071] In step S320, the number of symbols in the symbol number set is traversed, and the text to be processed is segmented according to the number of continuous line break symbols traversed to obtain a first target text segment set.

[0072] Exemplarily, a specific implementation of step S320 may include: determining a set of segmented text segments obtained by segmenting the text to be processed according to the number of symbols traversed last time, and obtaining segmented text segments; traversing the set of segmented text segments based on the number of symbols currently traversed, and when the text length of the traversed segmented text segment is less than or equal to the first preset text length, not processing the segmented text segment; when the text length of the traversed segmented text segment is greater than the first preset text length, segmenting the segmented text segment according to the number of symbols traversed, and updating the set of segmented text segments; obtaining the first target text segment set based on the set of segmented text segments updated according to the number of symbols traversed last time.

[0073] Taking the above-mentioned symbol quantity set {8, 6, 5, 4, 3, 2, 1} as an example, the first symbol quantity 8 in the symbol quantity set can be traversed first, and then the text to be processed is segmented according to 8 consecutive line feed symbols, such as obtaining the segmented text segment set C1 = {sub_text1, sub_text2, sub_text3, ..., sub_textN}, and then, continue to traverse the next symbol quantity 6 in the symbol quantity set, so as to segment the text segments in C1 according to 6 consecutive line feed symbols. Specifically, the text segments in C1 can be traversed, and when the text length of the text segment in C1 is less than or equal to the first preset text length, it is not processed. When the text length of the text segment in C1 is greater than the first preset text length, the text segment in C1 is continued to be segmented according to 6 consecutive line feed symbols, and the segmented text segments are used to replace the text segments in C1 until all text segments in C1 are traversed, thereby updating C1. Continue to traverse the next line break symbol number 5 in the symbol quantity set, and based on 5 consecutive line break symbols, traverse C1 obtained after the previous update based on 6 consecutive line break characters. When the traversed text segment length is greater than the first preset text length, further segment the traversed text segment according to 5 consecutive line break symbols, otherwise, do not process the traversed text segment, thereby realizing the iterative update of C1, and so on, until the length of the text segment in the updated C1 is less than or equal to the first preset text length, then stop updating C1, or until all the symbol quantities in the symbol quantity set are traversed, that is, until C1 is updated according to the last symbol quantity in the symbol quantity set, then stop updating C1.

[0074] Taking the example that the number of line break symbols currently traversed is 8 in the above-mentioned symbol number set {8, 6, 5, 4, 3, 2, 1}, the process of segmenting the text to be processed according to the number of 8 consecutive line break symbols can be: traverse the characters in the text to be processed, and when 8 consecutive line break symbols are traversed, split the text before and after the 8 consecutive line break symbols. For example, for a text to be processed, 8 consecutive line break symbols are traversed a total of 6 times, then based on 8 consecutive line break symbols, the text to be processed can be segmented into 7 text segments.

[0075] Continuing with the above-mentioned symbol set {8, 6, 5, 4, 3, 2, 1} as an example, first traverse the text to be processed, split the text to be processed according to 8 consecutive line feed symbols, and obtain the segmented text segment set C1 = {sub_text1, sub_text2, sub_text3, ..., sub_textN}, then traverse the next number of line feed symbols in the line feed symbol number set, that is, 6, and then traverse the segmented text segment set C1 = {sub_text1, sub_text2, sub_text3, ..., sub_textN}, when the text length of the traversed text segment is greater than the first preset text length, continue to segment the text segment according to 6 consecutive line feed symbols, after the segmentation is completed, continue to traverse the next text segment, when the text length of the traversed text segment is less than or equal to the first preset text length, do not process the text segment, and continue to traverse the next text segment, thereby realizing the update of the segmented text segment set C1.

[0076] For example, if the text lengths of the text segments sub_text1 and sub_text3 in C1={sub_text1, sub_text2, sub_text3, ..., sub_textN} are greater than the first preset text length, sub_text1 and sub_text3 can be further segmented according to 6 consecutive line feed symbols, such as sub_text11 and sub_text12 are obtained after sub_text1 is segmented, and sub_text31, sub_text32 and sub_text33 are obtained after sub_text3 is segmented. Then the segmented text segment set C1 is updated to C1=={sub_text1, sub_text2, sub_text3, ..., sub_textN}. 1, sub_text12, sub_text2, sub_text31, sub_text32, sub_text33, …, sub_textN}, and then continue to traverse the next number of symbols in the symbol quantity set, that is, 5, based on the number of 5 consecutive newline symbols, traverse the text segment in C1 updated last time, and determine whether to continue to split the text segment according to the 5 consecutive newline symbols according to the length of the traversed text segment, until the text segment in C1 updated last time is traversed, thereby realizing the re-update of the segmented text segment set C1, and continue to traverse the next number of symbols in the symbol quantity set, that is, 4, and so on, until the elements in the symbol quantity set are traversed and then end.

[0077] In other words, each time the segmented text segment set is updated, first traverse the elements in the symbol quantity set to determine the current number of segmentation symbols, then traverse the segmented text segments in the segmented text segment set obtained in the previous update, and determine whether the segmented text segment needs to be segmented based on the current number of segmentation symbols according to the text length of the segmented text segment. If necessary, segment the segmented text segment according to the current number of segmentation symbols, until the text segments in the segmented text segment set obtained in the previous update are traversed, and then continue to traverse the next element in the symbol quantity set, and repeat the above process until all the elements in the symbol quantity set are traversed.

[0078] In an exemplary embodiment, in order to improve efficiency, a mark can be added to the text segment whose text length has been determined to be less than the first preset text length. In this way, during subsequent traversal, there is no need to judge the text length again, and it can be directly determined that it does not need to be split again, so that the next text segment can be traversed directly.

[0079] Exemplarily, another specific implementation of step S320 may include: segmenting the text to be processed according to the maximum number of symbols in the symbol number set to obtain a set of text segments to be traversed; deleting the maximum number of symbols in the symbol number set to obtain a set of symbol numbers to be traversed; traversing the text segments to be traversed in the set of text segments to be traversed, and when the text length of the text segments to be traversed is greater than the first preset text length, traversing the number of symbols in the symbol number set to be traversed, and iteratively segmenting the text segments to be traversed according to the traversed number of symbols, until the text length of each sub-text segment to be traversed after segmentation is less than or equal to the first preset text length. After the text length or the elements in the set of symbol numbers to be traversed are traversed, traversal continues for the next text segment to be traversed. For the next text segment to be traversed, the number of symbols in the set of symbol numbers to be traversed is traversed again. The next text segment to be traversed is iteratively divided according to the traversed number of symbols until the text length of each sub-text segment to be traversed obtained after the division is less than or equal to the first preset text length or the number of symbols in the set of symbol numbers to be traversed is traversed. Then traversal continues for the next text segment to be traversed. The above process is repeated until the text segments to be traversed in the set of text segments to be traversed are completely traversed, thereby obtaining the first target text segment set.

[0080] For example, the text to be processed can be first divided into N text segments according to the maximum number of symbols, and then the N text segments are traversed in turn. When the text length of the traversed text segment is greater than the first preset length, the other numbers of symbols in the symbol quantity set are traversed in turn. For example, the traversed text segment is first divided according to the second number of symbols to obtain Q text segments. For each of the Q text segments, it is determined whether it needs to be divided according to the third number of symbols according to its text length, and so on, until the length of each divided text segment is less than or equal to the first preset text length, then continue to traverse the next text segment in the N text segments, and continue to perform iterative segmentation operations on it, until all N text segments are traversed, and finally the first target text segment set is obtained. That is to say, for each of the N text segments, it can be continuously segmented according to the number of other symbols in the symbol number set until the text length of the final segmented text segment is less than or equal to the first preset text length or until the symbol number set is completely traversed, and then the next text segment is traversed and continuously segmented according to the symbol number set.

[0081] As mentioned earlier, usually, a long text, such as different chapters in an article, may be separated by line breaks. A segment with more line breaks represents a large segment of text with complete semantics, that is, different chapters may be separated by more line breaks, and different sections in the same chapter may be separated by fewer line breaks, such as two chapters may be separated by 4 line breaks, and two sections in the same chapter may be separated by 2 line breaks, etc. Usually, different chapters introduce different content, that is, the content of different chapters is not very relevant.

[0082] In the present disclosure, the number of line break symbols is traversed from many to few to segment the text. Texts with large semantic differences can be segmented into different text segments according to a larger number of line break symbols, such as texts between different chapters into different text segments. Then, the contents of different sections in the same chapter can be segmented into different text segments according to a smaller number of line break symbols. In this way, texts with different semantics can be segmented into different text segments as much as possible, and texts with the same semantics can be segmented into the same text segment, thereby improving the semantic accuracy of text segmentation.

[0083] For example, Figure 4 A flowchart of another method for obtaining a first target text segment set in an exemplary embodiment of the present disclosure is shown. Figure 4 , the method may include steps S401 to S407. Wherein: in step S401, the space symbols between the line break symbols in the text to be processed are deleted to obtain the cleaned text; in step S402, the number of continuous line break symbols in the text to be processed is extracted; in step S403, the number of line break symbols is traversed in descending order; in step S404, the cleaned text is segmented according to the number of line break symbols traversed to obtain segmented text segments; in step S405, it is determined whether the length of the segmented text segment is less than the first preset text length, if so, it goes to step S403, otherwise it goes to step S406; in step S406, the segmented text segment is not processed; in step S407, after the number of line break symbols is traversed, the first target text segment is determined according to the segmented text segment obtained.

[0084] For example, the spaces between the continuous line break symbols in the to-be-processed text can be deleted first to obtain the cleaned text. Then, the number of continuous line break symbols in the cleaned text is extracted, and the number of extracted line break symbols is sorted to obtain a symbol number list line_feed_num_list = [N1, ..., Nn], and the symbol number list is traversed, that is, the line break symbols are traversed in order from large to small to segment the cleaned text. Specifically, after the cleaned text is segmented according to the number of continuous line break symbols currently traversed, N text segments are obtained, such as sub_text_list = [sub_text1, sub_text2, sub_text3, ..., sub_textN], and then the sub_text_list is traversed to determine the text length of each sub_text. If its text length is less than or equal to the first preset text length of the text, then this sub_text does not participate in the next round of text segmentation traversal, that is, there is no need to segment the sub_text according to the number of line break symbols traversed next time. If the text length of the traversed sub_text is greater than the first preset text length, it will continue to participate in the next round of text segmentation, that is, continue to segment the sub_text according to the number of line feed symbols traversed next time, and the sub_text participating in the next round of text segmentation will be divided into N text segments to obtain [sub_sub_text1, sub_sub_text2, ..., sub_sub_textN], and then merged into sub_text_list, until the line feed symbol number list line_feed_num_list is traversed completely, and the first target text segment is obtained according to the text segment obtained after the end.

[0085] Next, continue to refer to Figure 2 In step S220, based on the similarities between adjacent first target text segments in the first target text segment set, the first target text segments in the first target text segment set are grouped to obtain a text segment grouping result.

[0086] For example, Figure 5 A flowchart of a method for obtaining text segment grouping results in an exemplary embodiment of the present disclosure is shown, referring to Figure 5 , the method may include steps S510 to S540. Wherein:

[0087] In step S510, based on the similarity between adjacent first target text segments and the similarity threshold, the first target text segments in the first target text segment set are grouped to obtain a grouping processing result.

[0088] Exemplarily, the first target text segments in the first target text segment set may be vectorized to obtain text segment vectors, and the similarities between adjacent first target text segments may be calculated based on the text segment vectors.

[0089] Of course, the similarity between adjacent first target text segments may also be calculated in other ways, for example, using a pre-trained machine learning model to calculate the similarity between adjacent first target text segments, etc. This exemplary embodiment does not specifically limit this.

[0090] Among them, the first target text segment can be textembedding (text feature embedding) through the SentenceModel model of text2vec to obtain a text segment vector. Text2vec is an existing technology for converting text into vectors. It can convert a text into a set of numerical values ​​to represent the features of this text. Of course, the first target text segment can be vectorized by other methods to obtain the text segment vector of the first target text segment, and this exemplary embodiment does not make any special limitation on this.

[0091] After obtaining the text segment vector of the first target text segment, the similarity between the two first target segments can be calculated based on the text segment vector. For example, the cosine_similarity (cosine similarity) algorithm integrated by the sklearn machine learning library can be used to calculate the cosine similarity between the text segments based on the text segment vector. Cosine similarity is to evaluate their similarity by calculating the angle cosine value of two vectors, and cosine similarity draws the vector into the vector space according to the coordinate value, such as in the most common two-dimensional space, to evaluate the similarity between the two. Of course, the similarity between the text segments can also be calculated by other methods, and this exemplary embodiment is not particularly limited to this.

[0092] For example, after determining the similarities between adjacent first target text segments, a text segment similarity set, such as a text segment similarity list, may be generated, in which the similarities between adjacent first target text segments are recorded. For example, the first three data in the similarity list represent, from front to back, the similarity between the first first target text segment and the second first target text segment, the similarity between the second first target text segment and the third first target text segment, and the similarity between the third first target text segment and the fourth first target text segment.

[0093] In an exemplary implementation, the first target text segments in the first target text segment set may be grouped according to an initial similarity threshold. Based on this, exemplarily, a specific implementation of step S510 may include: when the similarity between adjacent first target text segments in the first target text segment set is greater than or equal to the initial similarity threshold, the two first target text segments may be allocated to different groups, otherwise, the two first target text segments are allocated to the same group, thereby achieving initial grouping of the first target text segments in the first target text segment set and obtaining an initial grouping processing result.

[0094] In other words, grouping based on text segment similarity mainly determines whether two text segments can be grouped in the same group based on the similarity between the text segments. Generally, the smaller the similarity distance (such as the cosine similarity distance mentioned above) between two text segments, the larger the similarity value, which proves that the semantic difference between the two text segments is smaller, and they can be assigned to the same group. The larger the similarity distance, the smaller the similarity value, which proves that the semantic difference between the two text segments is greater, and they cannot be assigned to the same group.

[0095] In step S520, the similarity threshold is updated successively in a decreasing manner, and for each group in the previous grouping processing result, the first target text segments in the group are grouped based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update, so as to update the grouping processing result until the preset condition is met.

[0096] For example, the similarity threshold may be gradually decreased according to a fixed difference or a variable difference to update the similarity threshold, such as decreasing by 0.1 each time or decreasing by 0.1 for the first time, decreasing by 0.2 for the second time based on the first decrease, decreasing by 0.3 for the third time based on the second decrease, etc.

[0097] In an exemplary embodiment, the role of the preset condition is to stop updating the similarity threshold to determine the final grouping processing result. The preset condition may include the similarity threshold decreasing to 0, the number of updates of the similarity threshold reaching a preset number, etc. Of course, the preset condition can also be set according to needs, and this exemplary embodiment does not make any special limitations on this.

[0098] Exemplarily, for each group in the previous grouping processing result, the first target text segments in the group are grouped based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update to update the grouping processing result, including: for each group in the previous grouping processing result, when the text length of the first target text segment in the group is greater than the second preset text length, the first target text segments in the group are grouped based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update; otherwise, the first target text segments in the group are not grouped to update the grouping processing result.

[0099] Exemplarily, when the text length of the first target text segments in the group is greater than the second preset text length, based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update, the first target text segments in the group are grouped, including: when the similarity between adjacent first target text segments in the group is less than or equal to the similarity threshold after the current update, determining that the adjacent first target text segments in the group belong to the same group; otherwise, determining that the adjacent first target text segments in the group belong to different groups.

[0100] For example, for each group in the previous grouping processing result, the total length of the first target text segments contained in the group can be calculated. If the total length of the first target text segments contained in the group is greater than the second preset text length, the first target text segments in the group are further grouped according to the similarity threshold after the current update, that is, the first target text segments with similarities less than or equal to the updated similarity threshold are grouped in the same group, and the first target text segments with similarities greater than the similarity threshold after the current update are grouped in different groups, until further grouping is completed for each group in each previous grouping processing result according to the similarity threshold after the current update, and then the similarity threshold is updated, and the process is repeated until the similarity threshold is reduced to 0.

[0101] For example, according to the first similarity threshold, the first target text segment in the first target text segment set is divided into 10 groups, and then the similarity threshold is updated to obtain the second similarity threshold. According to the second similarity threshold, the first target text segment in the group whose text length is greater than the second preset text length in the 10 groups in the first grouping is further grouped. If the text length of 2 of the 10 groups in the first grouping is greater than the second preset text length, these 2 groups are further divided into 2 groups according to the second similarity threshold. Then, after the current update, the grouping processing result has a total of There are 14 groups. Since the lengths of other groups have been determined to be less than or equal to the second preset text length, after updating the similarity threshold to obtain the third similarity threshold, it is only necessary to judge the text length of the 4 groups newly determined for the second time to determine whether they need to continue to be grouped according to the third similarity threshold, and so on. The iterative process of grouping the groups in the previous grouping again according to the updated similarity threshold is repeated until the similarity threshold is reduced to 0 or the total text length of all groups is less than or equal to the second preset text length.

[0102] In the present disclosure, by updating the similarity threshold in the direction of decreasing similarity threshold, the grouping is continuously updated iteratively, and the text segments with large semantic differences can be first divided into different groups, and then the text segments with similar semantics are further grouped according to a smaller similarity threshold, thereby improving the segmentation efficiency while ensuring the semantic accuracy of the segmented text. In addition, the present disclosure takes into account the length of the text segment when further grouping, so the segmentation length requirement is met while ensuring the semantic accuracy of the segmentation.

[0103] In an exemplary embodiment, the first preset text length and the second preset text length can be customized according to requirements.

[0104] In an exemplary application scenario, the text segment obtained after segmentation according to the text segmentation method disclosed in the present invention can be applied to the large language model training, based on which the second preset text length can be determined according to the input text length of the large language model. The size relationship between the first preset text length and the second preset text length can be freely configured, that is, the first preset text length can be greater than the second preset text length, and the first preset text length can also be less than the second preset text length, and this exemplary embodiment does not make any special restrictions on this.

[0105] In step S530, the text segment grouping result is obtained based on the last updated grouping processing result.

[0106] The last updated grouping processing result may be understood as the grouping processing result updated according to the last updated similarity threshold.

[0107] In an exemplary implementation, the last updated grouping processing result may be directly determined as the text segment grouping result.

[0108] In another exemplary implementation, the last updated grouping processing result may be cleaned to obtain the text segment grouping result. Figure 6 A flow chart of a method for cleaning a grouping processing result to obtain a text segment grouping result in an exemplary embodiment of the present disclosure is shown. Figure 6 , the method may include steps S610 to S630.

[0109] in:

[0110] In step S610, for each group in the last updated grouping processing result, the first target text segment in the group is concatenated to obtain a second target text segment.

[0111] For example, for each group in the final grouping processing result, the first target text segments in the group are sequentially spliced ​​to obtain the second target text segment. If there are 100 groups, there are 100 second target text segments.

[0112] In step S620, for each second target text segment, a feature value corresponding to the text segment quality feature in the second target text segment is obtained, and the feature value is input into a text quality prediction model to determine a text quality score of the second target text segment.

[0113] In an exemplary embodiment, the text segment quality features include one or more of preset keywords, text segment length, proportion of punctuation marks in the text segment, proportion of pause words in the text segment, and text perplexity.

[0114] The preset keywords can be determined according to the needs, and the preset keywords can include texts that are irrelevant to the main content of the segmented text. For example, for a book, the publisher, author, references, publisher contact number, etc. can be considered as noise that is irrelevant to the main content of the book, so the preset keywords can include publisher, author, references, publisher contact number, etc.

[0115] The length of the text can also be used to evaluate the quality of the text segment obtained after segmentation. Generally speaking, the longer the text is, the more likely it is to be a high-quality text. The higher the proportion of punctuation marks or pause words in a text segment, the worse the quality of the segmented text segment.

[0116] The perplexity of a text is used to measure how well a probability distribution or probability model predicts a sample. It can also be used to compare two probability distributions or probability models. In the present disclosure, it can be used to measure the fluency of a sentence text. The higher the perplexity of the text, the worse the semantic fluency of the text, and the more likely the text is a low-quality text.

[0117] It should be noted that the text quality feature may also include other indicators that can measure the quality of the segmented text segments, and this exemplary embodiment does not specifically limit this.

[0118] Exemplarily, texts with different text quality feature values ​​may be marked with text quality scores in advance to generate training data, and a machine learning model may be trained based on the training data to obtain a text quality prediction model. For example, a linear regression function may be fitted based on the training data to obtain a text quality prediction model. Of course, other machine learning models may also be used, such as a neural network model, etc., and this exemplary embodiment does not specifically limit this.

[0119] For example, for each second target text segment, the text quality feature value of the second target text segment can be extracted according to the text quality features, such as extracting preset keywords included in the second target text segment, extracting the text length of the second target text segment, determining the proportion of punctuation marks in the second target text segment in the entire text segment, determining the proportion of pause words in the second target text segment in the entire text segment, determining the text perplexity of the second target text segment, etc. The extracted and determined text quality feature values ​​are input into the text quality prediction model to obtain the text quality score of the second target text segment.

[0120] Next, in step S630, the second target text segments whose text quality scores are less than the text quality threshold are filtered out, and the text segment grouping result is obtained based on the filtered second target text segments.

[0121] For example, the second target text segments with text quality scores less than the text quality threshold can be eliminated, and the text segment grouping result can be obtained based on the remaining second target text segments. The text quality threshold can be customized according to the requirements.

[0122] Usually, there may be noise in the text, and these noises will most likely be segmented into independent groups. By cleaning these noise data through the above-mentioned steps S610 to S630, the quality of the segmented text segments can be guaranteed.

[0123] In an exemplary embodiment, the above-mentioned grouping process can also be understood as bucketing process, where one group is equivalent to an independent bucket. Similarly, based on the similarity distance between adjacent first target text segments, it can also be determined whether the two can be placed in the same bucket. As mentioned above, the smaller the similarity distance, the smaller the text semantic difference between the two, and they can be allocated to the same bucket. The larger the similarity distance, the greater the text semantic difference, and they cannot be allocated to the same bucket.

[0124] In this disclosure, length_bucket can be set as the length of the text in the bucket, that is, the maximum text length of each bucket, init_text_sim_distance is the initial similarity threshold, and step is the step size of each iteration. The step should not be set too large. The smaller the step size, the more iterations are required, but it is more likely that text segments with the same semantics will be grouped into the same bucket. Based on this, Figure 7 A flowchart of a method for bucketing a text segment in an exemplary embodiment of the present disclosure is shown. Figure 7 , the method may include steps S710 to S750.

[0125] In step S710, the similarity set of the text segments to be bucketed is traversed.

[0126] For example, the text segments to be bucketed may include the above-mentioned first target text segments, and the similarity set of the text segments to be bucketed may be determined based on the above-mentioned text segment similarity list that records the similarities between adjacent first target text segments. If before bucketing, the text segments to be bucketed include all the first target text segments, the current similarity set of the text segments to be bucketed may be the above-mentioned text segment similarity list, and after bucketing, the current similarity set of the text segments to be bucketed may be a subset of the above-mentioned text segment similarity list. If 10 buckets are obtained after the first bucketing, and the first bucket needs to be bucketed again, and the first bucket includes the first 9 first target text segments in the first target text segment set, then for the first bucket, the similarity set of the text segments to be bucketed is a subset of the first 8 elements in the above-mentioned text segment similarity list.

[0127] In step S720, the text segments to be bucketed are bucketed according to the traversed similarities and the current text segment similarity threshold and based on the text segment bucketing strategy.

[0128] In an exemplary embodiment, the current text segment similarity threshold text_sim_distance can be determined by the difference between the current initial similarity threshold init_text_sim_distance and the iteration step, that is, text_sim_distance = init_text_sim_distance-step. The initialized initial similarity threshold is usually set relatively large, and the larger the value, the more likely it is to merge text segments with the same semantics into the same bucket.

[0129] For example, the text segment bucketing strategy may include, when the traversed similarity is greater than the current text segment similarity threshold, grouping the two adjacent first target text segments indicated by the similarity into different buckets; otherwise, grouping the two adjacent first target text segments indicated by the similarity into the same bucket.

[0130] In step S730, the text segment similarity threshold is updated, and each bucket after the previous bucket processing is traversed.

[0131] Exemplarily, the text segment similarity threshold can be updated in the direction of decreasing the initial similarity threshold according to the step length to obtain the current text segment similarity threshold. In other words, the value of the initial similarity threshold init_text_sim_distance can be reduced first, and then the current text segment similarity threshold text_sim_distance can be calculated according to the determination method of the current similarity threshold text_sim_distance=init_text_sim_distance-step, so as to achieve the update of the text segment similarity threshold.

[0132] In step S740, it is determined whether the total length of the text segments in each bucket is greater than the bucket text length. If so, the process goes to step S710; otherwise, the process goes to step S750.

[0133] In step S750, no further bucketing process is performed on the text segment in the bucket.

[0134] For example, after the text segment similarity threshold is updated to obtain the current text segment similarity threshold, each bucket after the previous bucketing process can be traversed to determine the total length of the text segments in each bucket. If the total length of the text segments in the bucket is greater than the bucket text length length_bucket, the text segments in the bucket enter the next round of iteration, that is, it is necessary to continue to bucket the text segments in the bucket based on the text segment bucketing strategy and the updated current text segment similarity threshold according to the text segment similarity threshold updated in step S730. If the total length of the text segments in the bucket is less than or equal to the bucket text length length_bucket, the text segments in the bucket enter the next round of iteration. length_bucket, the text segment in the bucket will not be included in the subsequent iterations, that is, the text segment in the bucket will not be further bucketed. At the same time, a mark can be added to the bucket, which can be used to indicate that the text segment in the bucket will not be included in the subsequent iterations. In this way, in the subsequent process, there is no need to compare the total length of the text segment in the bucket with the bucket text length to determine whether the text segment in the bucket needs to be further bucketed. Instead, it can be directly determined based on the mark that the text segment in the bucket does not need to be bucketed again. The above process is repeated until the iterative value text_sim_distance is 0.

[0135] Through the above steps S710 to S750, the text can be segmented based on the text segment similarity and the text segment length. The segmentation based on the text segment similarity can ensure the semantic accuracy of the segmented text segment. On the basis of segmentation based on the text segment similarity, the length of the text segment is taken into consideration, so that the length of the segmented text segment can meet the business scenario requirements. Therefore, the semantic accuracy of the segmentation can be ensured while the length of the segmented text segment can meet the business scenario requirements.

[0136] Continue to refer Figure 2 In step S230, the segmentation result of the text to be processed is determined according to the text segment grouping result.

[0137] In an exemplary implementation, the text segment grouping result obtained in step S220 can be directly determined as the segmentation result of the text to be processed. If the text segment grouping result includes 100 groups, the segmentation result of the text to be processed is segmented into 100 text segments, and the 100 text segments are respectively text segments in the 100 groups.

[0138] As mentioned above, the text segmentation method in the present disclosure can be applied to the training of a large language model. Based on this, in another exemplary embodiment, the determining the segmentation result of the text to be processed based on the text segment grouping result includes: when the total length of the text segments in K consecutive groups is less than or equal to the second preset text length, merging the K consecutive groups to obtain a merged text segment grouping result; determining the segmentation result of the text to be processed based on each grouped text segment in the merged text segment grouping result; wherein the K consecutive groupings minimize the sum of the differences between the total length of the text segments in each group in the merged text segment grouping result and the second preset text length.

[0139] Taking the training length of the large language model as the second preset text length as an example, although the total length of the text segments in each group in the text segment grouping results meets the requirement of being less than or equal to the second preset text length, the difference between the total length of the text segments in each group and the second preset text length may be large. Therefore, if the total length of the text segments in consecutive groups in the text segment grouping results is still less than or equal to the second preset text length, the consecutive groups can be merged so that the sum of the differences between the length of each segmented text segment obtained after the merger and the second preset text length is minimized.

[0140] For example, the text segment grouping results of a certain text to be processed include 100 groups, among which, for the continuous groups 5, 6, 7, 8, 9, and 10 in the group processing results, the total length of the text segments of groups 5, 6, and 7, the total length of the text segments of groups 6, 7, 8, and 9, and the total length of the text segments of groups 9 and 10 are all less than or equal to the second preset text length. However, the sum of the differences between the total length of the text segments in each group in the text segment grouping results obtained after merging groups 5, 6, and 7 and merging groups 9 and 10 and the second preset text length is smaller than the sum of the differences between the total length of the text segments in each group in the text segment grouping results obtained after merging groups 6, 7, 8, and 9 and the second preset text length. Therefore, the continuous K groups include two cases, namely continuous groups 5, 6, and 7 and continuous groups 9 and 10. Therefore, groups 5, 6, and 7 are merged into one group, and groups 9 and 10 are merged into one group. In this way, the text to be processed is divided into 97 text segments, and these 97 text segments correspond to the text segments in the 97 groups obtained after the merger, that is, each group corresponds to a segmented text segment. In other words, all the text segments in each group can be sequentially spliced ​​to obtain a final segmented text segment.

[0141] For example, Figure 8 A flowchart of another text segmentation method in an exemplary embodiment of the present disclosure is shown. Figure 8 , the method may include steps S801 to S806. Wherein:

[0142] In step S801, the text to be processed is segmented according to line break symbols to obtain a first text segment.

[0143] For example, step S801 mainly divides the long text into fine-grained text segments, thereby preparing for calculating the similarity between the text segments in the subsequent step S803. Among them, the method of dividing the text to be processed according to the line break symbol has been explained in the previous relevant part, and will not be repeated here.

[0144] In step S802, the first text segment is vectorized based on the text2vec model to obtain a first text segment vector list.

[0145] In step S803, the similarities between two adjacent first text segment vectors in the first text segment vector list are calculated in sequence to obtain a text segment similarity list.

[0146] In step S804, the first text segment is bucketed based on the text segment similarity list to obtain a text segment bucketing processing result.

[0147] The specific implementation method of bucketing the first text segment based on the text segment similarity list can refer to the above Figure 7 The text segment bucketing method shown is not described in detail here.

[0148] In step S805, data cleaning is performed on the text segment bucketing results based on the text quality features to remove noise.

[0149] The specific implementation method of data cleaning based on the text quality features of the text segment bucket results can also refer to the above Figure 6 The relevant contents in will not be repeated here.

[0150] In step S806, buckets in the text segment bucket processing result are merged according to the text length to generate a text segment segmentation result.

[0151] As mentioned above, the text segmentation method in the present disclosure can apply a large language model, and the text segments in the merged buckets after segmentation may not all meet the text length required for large language model training. Therefore, if the total length of the text in K consecutive buckets is still less than or equal to the text length required for large language model training, the consecutive buckets are merged to minimize the difference between the length of each text segment finally segmented and the text length required for large language model training.

[0152] In the present disclosure, a long text is first segmented into fine-grained text segments based on common symbols, such as punctuation marks or line breaks, and then the aforementioned segmented text segments are merged based on similarity, so that texts with the same semantics are segmented into the same text segment as much as possible, and texts with different semantics are segmented into different text segments as much as possible, thereby improving the semantic accuracy after text segmentation, and most text segments can be used to calculate similarity, so the text segmentation method in the present disclosure has strong versatility. At the same time, when segmenting text based on similarity, the present disclosure also takes into account the length of the text. Users can configure the length of the segmented text according to their needs to meet different business scenarios, which further improves the versatility of the text segmentation method in the present disclosure.

[0153] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0154] Furthermore, the exemplary embodiment of the present disclosure also provides a text segmentation device. Fig. 9 As shown, the text segmentation device includes the following program modules: a symbol segmentation module 910, which is configured to segment the text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; a similarity grouping module 920, which is configured to group the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments in the first target text segment set to obtain a text segment grouping result; a segmentation result determination module 930, which is configured to determine the segmentation result of the text to be processed according to the text segment grouping result.

[0155] In an exemplary implementation, based on the foregoing embodiments, when the general symbols include the punctuation marks, the symbol segmentation module 910 may be specifically configured to segment the text to be processed according to the punctuation marks in the text to be processed to obtain a first target text segment set.

[0156] In an exemplary embodiment, based on the foregoing embodiment, when the common symbol includes the line break symbol, the symbol segmentation module 910 can be specifically configured to: segment the text to be processed according to the number of consecutive line break symbols in the text to be processed to obtain a first target text segment set.

[0157] In an exemplary embodiment, based on the aforementioned embodiment, segmenting the text to be processed according to the number of consecutive line break symbols in the text to be processed to obtain a first target text segment set includes: determining the number of consecutive line break symbols in the text to be processed, sorting the number of symbols in descending order to obtain a symbol number set; traversing the number of symbols in the symbol number set, segmenting the text to be processed according to the number of consecutive line break symbols traversed to obtain a first target text segment set.

[0158] In an exemplary embodiment, based on the aforementioned embodiment, the traversing the number of symbols in the symbol quantity set and segmenting the text to be processed according to the number of consecutive line break symbols traversed to obtain the first target text segment set includes: determining the segmented text segment set based on the segmented text segments obtained after segmenting the text to be processed according to the number of consecutive line break symbols traversed last time; traversing the segmented text segments in the segmented text segment set based on the number of symbols currently traversed, and when the text length of the traversed segmented text segment is greater than the first preset text length, segmenting the segmented text segment according to the number of consecutive line break symbols currently traversed to update the segmented text segment set; obtaining the first target text segment set based on the segmented text segment set updated according to the number of symbols traversed last time.

[0159] In an exemplary embodiment, based on the aforementioned embodiment, the similarity grouping module 920 can be specifically configured as follows: based on the similarity between adjacent first target text segments and a similarity threshold, the first target text segments in the first target text segment set are grouped to obtain a grouping processing result; the similarity threshold is updated in a decreasing manner according to the similarity threshold, and for each group in the previous grouping processing result, the first target text segments in the group are grouped based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update to update the grouping processing result until a preset condition is met; and the text segment grouping result is obtained based on the grouping processing result of the last update.

[0160] In an exemplary embodiment, based on the aforementioned embodiment, for each group in the previous grouping processing result, based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update, the first target text segments in the group are grouped to update the grouping processing result, including: for each group in the previous grouping processing result, when the text length of the first target text segment in the group is greater than the second preset text length, the first target text segments in the group are grouped based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update; otherwise, the first target text segments in the group are not grouped to update the grouping processing result.

[0161] In an exemplary embodiment, based on the aforementioned embodiment, when the text length of the first target text segment in the group is greater than the second preset text length, based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update, the first target text segments in the group are grouped, including: when the similarity between adjacent first target text segments in the group is less than or equal to the similarity threshold after the current update, determining that the adjacent first target text segments in the group belong to the same group; otherwise, determining that the adjacent first target text segments in the group belong to different groups.

[0162] In an exemplary embodiment, based on the aforementioned embodiment, the text segment grouping result obtained based on the last updated grouping processing result includes: for each group in the last updated grouping processing result, splicing the first target text segment in the group to obtain a second target text segment; for each second target text segment, obtaining a feature value corresponding to a text segment quality feature in the second target text segment, inputting the feature value into a text quality prediction model, and determining a text quality score of the second target text segment; filtering out the second target text segments whose text quality scores are less than a text quality threshold, and obtaining the text segment grouping result based on the filtered second target text segments; wherein the text segment quality features include one or more of preset keywords, text segment length, the proportion of punctuation marks in the text segment, the proportion of pause words in the text segment, and text perplexity.

[0163] In an exemplary embodiment, based on the aforementioned embodiment, the segmentation result determination module 930 can be specifically configured as follows: when the total length of the text segments in K consecutive groups is less than or equal to the second preset text length, merge the K consecutive groups to obtain a merged text segment grouping result; determine the segmentation result of the text to be processed based on the text segments in each group in the merged text segment grouping result; wherein the K consecutive groups minimize the sum of the differences between the total length of the text segments in each group in the merged text segment grouping result and the second preset text length.

[0164] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0165] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0166] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0167] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the text segmentation method described above is implemented.

[0168] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0169] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0170] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, Python, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0171] The computer program can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. The electronic device can convert the signal carrying the computer program into a digital signal, and then run the computer program. When the computer program is running on the electronic device, its code is used to make the electronic device execute (more specifically, it can make the processor of the electronic device execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned text segmentation method, which includes the following steps: segmenting the text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break symbols; based on the similarity between adjacent first target text segments in the first target text segment set, the first target text segments in the first target text segment set are grouped to obtain a text segment grouping result; according to the text segment grouping result, the segmentation result of the text to be processed is determined.

[0172] The above method steps are executed by a computer program. On the one hand, after the text is segmented based on the common symbols, the text segments are grouped according to the similarity of the text segments to ensure that texts with the same semantics are segmented into the same text segment and texts with different semantics are segmented into different text segments, thereby improving the semantic accuracy of text segmentation; on the other hand, the common symbols and text segment similarity are applicable to most texts, thus ensuring the universality of the text segmentation method.

[0173] The exemplary embodiments of the present disclosure also provide an electronic device, such as the terminal device 110 or the server 120 described above. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0174] Reference below Fig.10 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig.10 The electronic device 1000 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0175] like Fig.10 As shown, the electronic device 1000 may include: a processor 1010 , a memory 1020 , a bus 1030 , an I / O (input / output) interface 1040 , a network adapter 1050 , and a display 1460 .

[0176] The memory 1020 may include a volatile memory, such as a RAM 1021, a cache unit 1022, and may also include a non-volatile memory, such as a ROM 1023. The memory 1020 may also include one or more program modules 1024, such program modules 1024 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 1024 may include each module in the above-mentioned device.

[0177] The processor 1010 may include one or more processing units. For example, the processor 1010 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0178] Processor 1010 can be used to execute executable instructions stored in memory 1020, such as the above-mentioned text segmentation method, which includes the following steps: segmenting the text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; based on the similarity between adjacent first target text segments in the first target text segment set, grouping the first target text segments in the first target text segment set to obtain a text segment grouping result; and determining the segmentation result of the text to be processed according to the text segment grouping result.

[0179] The above method steps are executed by a computer program. On the one hand, after the text is segmented based on the common symbols, the text segments are grouped according to the similarity of the text segments to ensure that texts with the same semantics are segmented into the same text segment and texts with different semantics are segmented into different text segments, thereby improving the semantic accuracy of text segmentation; on the other hand, the common symbols and text segment similarity are applicable to most texts, thus ensuring the universality of the text segmentation method.

[0180] The bus 1030 is used to realize the connection between different components of the electronic device 1000, and may include a data bus, an address bus, and a control bus.

[0181] The electronic device 1000 can communicate with one or more external devices 1100 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 1040 .

[0182] The electronic device 1000 can communicate with one or more networks through the network adapter 1050. For example, the network adapter 1050 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 1050 can communicate with other modules of the electronic device 1000 through the bus 1030.

[0183] The electronic device 1000 may display a graphical user interface through the display 1060 , such as an interface for displaying text segment grouping results.

[0184] although Fig.10 Not shown, other hardware and / or software modules may also be provided in the electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0185] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0186] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0187] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A text segmentation method, characterized in that: include: Segment the text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; Based on the similarity between adjacent first target text segments and the similarity threshold, the first target text segments in the first target text segment set are grouped to obtain a grouping processing result; The similarity threshold is updated successively in a decreasing manner, and for each group in the previous grouping processing result, when the text length of the first target text segment in the group is greater than the second preset text length, the first target text segments in the group are grouped based on the similarities between adjacent first target text segments in the group and the similarity threshold after the current update; otherwise, the first target text segments in the group are not grouped, so as to update the grouping processing result until the preset condition is met; Based on the last updated grouping processing result, the text segment grouping result is obtained; Determining the segmentation result of the text to be processed according to the text segment grouping result; The grouping of the first target text segments in the group based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update includes: When the similarity between adjacent first target text segments in the group is less than or equal to the similarity threshold after the current update, it is determined that the adjacent first target text segments in the group belong to the same group; otherwise, it is determined that the adjacent first target text segments in the group belong to different groups.

2. The text segmentation method according to claim 1, characterized in that: In the case where the universal symbols include the punctuation marks, segmenting the to-be-processed text according to the universal symbols to obtain a first target text segment set includes: The text to be processed is segmented according to punctuation marks in the text to be processed to obtain a first target text segment set.

3. The text segmentation method according to claim 1, characterized in that: In the case where the universal symbol includes the line break symbol, segmenting the to-be-processed text according to the universal symbol to obtain a first target text segment set includes: The text to be processed is segmented according to the number of continuous line break symbols in the text to be processed to obtain a first target text segment set.

4. The text segmentation method according to claim 3, characterized in that: The segmenting of the text to be processed according to the number of continuous line break symbols in the text to be processed to obtain a first target text segment set includes: Determine the number of consecutive line break symbols in the text to be processed, and sort the number of symbols in descending order to obtain a set of symbol numbers; The number of symbols in the symbol number set is traversed, and the text to be processed is segmented according to the number of continuous line break symbols traversed to obtain a first target text segment set.

5. The text segmentation method according to claim 4, characterized in that: The traversing the number of symbols in the symbol number set, segmenting the text to be processed according to the number of continuous line break symbols traversed, and obtaining the first target text segment set includes: The segmented text segments obtained by segmenting the to-be-processed text according to the number of continuous line break symbols traversed in the previous time are used to determine a segmented text segment set; Based on the number of symbols currently traversed, the segmented text segments in the segmented text segment set are traversed, and when the text length of the traversed segmented text segment is greater than the first preset text length, the segmented text segment is segmented according to the number of continuous line break symbols currently traversed, so as to update the segmented text segment set; The first target text segment set is obtained based on the segmented text segment set updated according to the number of symbols traversed last time.

6. The text segmentation method according to claim 1, characterized in that: The text segment grouping result obtained based on the last updated grouping processing result includes: For each group in the grouping processing result updated last time, concatenate the first target text segments in the group to obtain a second target text segment; For each second target text segment, obtaining a feature value corresponding to a text segment quality feature in the second target text segment, inputting the feature value into a text quality prediction model, and determining a text quality score of the second target text segment; Filter out the second target text segments whose text quality scores are less than the text quality threshold, and obtain the text segment grouping result based on the filtered second target text segments; The text segment quality features include one or more of preset keywords, text segment length, proportion of punctuation marks in the text segment, proportion of pause words in the text segment, and text perplexity.

7. The text segmentation method according to any one of claims 1 to 6, characterized in that: Determining the segmentation result of the to-be-processed text according to the text segment grouping result includes: When the total length of the text segments in the K consecutive groups is less than or equal to the second preset text length, merging the K consecutive groups to obtain a merged text segment grouping result; Determining the segmentation result of the to-be-processed text according to the text segments in each group in the merged text segment grouping result; The K consecutive groups are such that the sum of the differences between the total length of the text segments in each group in the merged text segment grouping result and the second preset text length is minimized.

8. A text segmentation device, characterized in that: include: A symbol segmentation module is configured to segment the text to be processed according to common symbols to obtain a first target text segment set, wherein the common symbols include one or more of punctuation marks and line break marks; The similarity grouping module is configured to group the first target text segments in the first target text segment set based on the similarity between adjacent first target text segments and the similarity threshold, obtain a grouping processing result, update the similarity threshold one by one in a decreasing manner according to the similarity threshold, and for each group in the previous grouping processing result, if the text length of the first target text segment in the group is greater than the second preset text length, group the first target text segments in the group based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update, otherwise, do not group the first target text segments in the group, so as to update the grouping processing result until the preset condition is met; Based on the last updated grouping processing result, the text segment grouping result is obtained; A segmentation result determination module is configured to determine the segmentation result of the to-be-processed text according to the text segment grouping result; The grouping of the first target text segments in the group based on the similarity between adjacent first target text segments in the group and the similarity threshold after the current update includes: When the similarity between adjacent first target text segments in the group is less than or equal to the similarity threshold after the current update, it is determined that the adjacent first target text segments in the group belong to the same group; otherwise, it is determined that the adjacent first target text segments in the group belong to different groups.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the text segmentation method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text segmentation method as described in any one of claims 1 to 7.

Images