Dialogue text segmentation method, device, storage medium and program product

Abstract

The present application provides a method, device, storage medium and program product for segmenting a conversation text, the method comprising: performing noise reduction processing on the original conversation text to obtain the corresponding noise-removed conversation text; extracting the corresponding first segmentation point set and second segmentation point set from the original conversation text and the noise-removed conversation text respectively; merging the first segmentation point set and the second segmentation point set to obtain the target segmentation point set; segmenting the original conversation text based on the target segmentation point set. Thus, by merging the two segmentation point sets and taking advantage of the complementary advantages of the two information sources, the original conversation text and the noise-removed conversation text, multi-perspective data support is provided, making the extraction of segmentation points more comprehensive and accurate.

Description

Technical Field

[0001] The present application relates to the technical field of text processing, and in particular to a conversation text segmentation method, device, storage medium and program product. Background Art

[0002] In today's era of digital information explosion, conversation data is growing in massive quantities. As a key link in the field of natural language processing, conversation text segmentation technology plays an extremely important cornerstone role in extracting valuable information from conversations.

[0003] Conversation text segmentation methods can be mainly divided into unsupervised text segmentation algorithms and supervised text segmentation algorithms. Unsupervised text segmentation methods mainly rely on the statistical characteristics and language structure of the text itself, without the need for a large amount of labeled data for training. Supervised text segmentation methods use a large amount of conversation text data with labeled segmentation boundaries for training, thereby learning a model that can predict segmentation points.

[0004] However, in unsupervised text segmentation methods, the lack of correct labeled data as a guide leads to relatively poor final text segmentation results. Simply using text similarity to calculate the text cutting position is basically unusable in some conversation scenarios. For example, in a conference conversation scenario, the segmented paragraphs have no obvious topic differences.

[0005] In supervised text segmentation methods, the segmentation effect can be effectively improved, but it cannot handle the problems of alternating topics, invalid topics and colloquialism in dialogue data. Existing models can extract correct results in obvious topic segmentation locations, but cannot achieve good results in some ambiguous and topic-nested situations. In addition, since different types of dialogue scenarios have different definitions of topics, existing supervised algorithms are also difficult to achieve universal results in multiple dialogue scenarios.

[0006] The industry has not yet proposed a better solution to the above problems. Summary of the invention

[0007] The present application provides a conversation text segmentation method, device, storage medium and program product, which are used to at least solve the problem that existing text segmentation algorithms cannot be better applied in multi-conversation scenarios.

[0008] In a first aspect, an embodiment of the present application provides a method for segmenting a conversation text, comprising: performing noise reduction processing on an original conversation text to obtain a corresponding denoised conversation text; extracting a corresponding first segmentation point set and a second segmentation point set from the original conversation text and the denoised conversation text, respectively; merging the first segmentation point set and the second segmentation point set to obtain a target segmentation point set; and segmenting the original conversation text based on the target segmentation point set.

[0009] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the conversation text segmentation method of any embodiment of the present application.

[0010] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the conversation text segmentation method of any embodiment of the present application are implemented.

[0011] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the conversation text segmentation method of any embodiment of the present application.

[0012] The beneficial effects of the embodiments of the present application are: The first segmentation point set and the second segmentation point set are extracted from the original conversation text and the denoised conversation text respectively, taking into full account the impact of different data processing methods on segmentation point identification. The segmentation points in the original conversation text can capture the obvious boundaries based on the language structure, while the denoised text can make up for the implicit boundaries in the original data that are missed due to the influence of noise. By merging these two segmentation point sets, the complementary advantages of the two information sources can be utilized, providing multi-perspective data support, making the extraction of segmentation points more comprehensive and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A flowchart showing an example of a method for segmenting a conversation text according to an embodiment of the present application is shown; Figure 2 A schematic diagram of the architecture of an example of a text segmentation model according to an embodiment of the present application is shown; Figure 3 An operation flow chart showing an example of performing noise reduction processing on an original conversation text according to an embodiment of the present application is shown; Figure 4 A schematic diagram showing the comparative effect of an example of a sentence before and after noise removal is shown; Figure 5 An operation flow chart of an example of extracting a segmentation point set according to an embodiment of the present application is shown; Figure 6 An operation flow chart of an example of merging a set of segmentation points according to an embodiment of the present application is shown; Figure 7 A schematic diagram showing the effect of an example of segmentation point density merging based on a clustering algorithm according to an embodiment of the present application is shown; Figure 8 An experimental simulation diagram showing an example of the influence of different search distances on the final segmentation effect according to an embodiment of the present application; Fig. 9 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be noted that in the unsupervised text segmentation algorithm, unsupervised text segmentation is implemented based on similarity measurement. For example, the three algorithms C99, TopicTiling and LCSeg use sentence similarity, LDA (Latent Dirichlet Allocation) and vocabulary chain to segment text; SITS model and MCMC model use hierarchical Bayesian model to segment dialogue. Early researchers used Hidden Markov Model (HMM) to cluster text fragments obtained from topic boundary information. With the emergence of pre-trained models, related studies use BERT (Bidirectional Encoder Representations from Transformers, pre-trained model) model to judge the topic relevance or coherence of discourse, and some researchers use neighboring discourse matching and pseudo segmentation to enable the model to learn topic-aware discourse representation.

[0017] In supervised text segmentation algorithms, with the popularity of deep neural networks and the emergence of large-scale annotated text segmentation data, supervised models based on deep neural networks began to appear. The earliest researchers contributed a large-scale text segmentation dataset and used Bi-LSTM to predict segment boundaries. Later, some researchers proposed using LSTM to predict sentence topics and merge them into segments through the SECTOR model. Some researchers proposed S-LSTM for text segmentation, adding additional coherence auxiliary tasks to enhance the segmentation performance of hierarchical attention Bi-LSTM. Some scholars also proposed using hierarchical Transformer networks to extract sentence features and then perform segmentation. Some researchers also pointed out that the fuzziness of dialogue segment boundaries and annotation noise will affect the segmentation effect and proposed a neighborhood smoothing method to deal with segment boundary noise and fuzziness.

[0018] Therefore, for the text segmentation task, researchers have proposed a variety of topic segmentation methods, which are divided into unsupervised methods and supervised methods. Unsupervised methods are mainly based on the characteristic that topic changes bring about vocabulary changes. They use a variety of similarity calculation algorithms to measure the differences between adjacent paragraphs. The points with large differences are set as segmentation points (TextTiling); or clustering algorithms are used to cluster texts, and texts under the same topic are classified into one category. This type of method usually ignores the impact of a large amount of noise, invalid text, and discrete information contained in the dialogue text on the text semantics. In addition, this type of algorithm does not have the correct human labels as a guide, resulting in unsatisfactory results in the end, and most of the extracted paragraph positions are wrong.

[0019] In addition, with the emergence of large-scale manually annotated datasets, research on text segmentation has gradually turned to supervised methods. Supervised methods generally predict whether each sentence is a segmentation point, thereby obtaining paragraph segmentation. Supervised algorithms have many limitations due to the uniqueness of conference conversations. First, different speakers may discuss multiple topics, which usually present a complex structure that alternates and nests with each other; second, the unstructured and colloquial characteristics of conversation texts, as well as the inaccuracy and information loss of speech transcriptions increase the difficulty of identifying topic boundaries; finally, different types of conversations or meetings (such as academic conferences, business conferences, government conferences, etc.) have significant differences in topic structure and discussion methods. Therefore, effective conference conversation topic segmentation technology needs to be able to identify and separate complex topic structures, be sufficiently robust to noisy text, and have high versatility and adaptability to different types of conference conversations to provide accurate conversation segmentation.

[0020] In view of this, Figure 1 A flowchart of an example of a method for segmenting a conversation text according to an embodiment of the present application is shown.

[0021] like Figure 1 As shown, in step S110, the original dialogue text is subjected to noise reduction processing to obtain the corresponding noise-removed dialogue text.

[0022] It should be noted that the noise reduction processing methods can be diverse to achieve text preprocessing, noise detection and denoising. In some examples, noise information can be defined as the unstructured part of the conversation, such as redundant stop words, modal particles, repeated content, verbal expressions, etc., and these irrelevant or redundant information can be identified and removed by using NLP (Natural Language Processing) technology, such as dictionary-based filtering, rule-based grammar parsing, and machine learning-based noise detection models. In other examples, the core content of the conversation can be screened out through part-of-speech tagging, syntactic analysis and other technologies, eliminating meaningless or off-topic parts and retaining valid information in the conversation.

[0023] In step S120, a corresponding first segmentation point set and a corresponding second segmentation point set are extracted from the original dialogue text and the noise-removed dialogue text respectively.

[0024] Here, the segmentation point extraction method can be diversified, such as based on an unsupervised text segmentation algorithm or a supervised text segmentation algorithm to identify potential segmentation points in the text. In addition, the segmentation point extraction method for the original dialogue text can be the same as or different from the segmentation point extraction method for the denoised dialogue text, which is not limited here.

[0025] It should be noted that the second segmentation point is obtained by performing segmentation point extraction on the denoised dialogue text, which can avoid the influence of noise in the original text and retain the contextual information after denoising. The structural nature of the denoised text helps to accurately extract more implicit and complex topic segmentation points.

[0026] In step S130 , the first segmentation point set and the second segmentation point set are merged to obtain a target segmentation point set.

[0027] In some implementations, each first segmentation point and each second segmentation point are sorted in text order, and if the segmentation points of the two overlap, one of them is retained. In addition, a merging threshold can be set, and when the distance between the first segmentation point and the second segmentation point is too small, only one segmentation point can be retained to remove redundancy.

[0028] Preferably, during the merging process, a semantic similarity evaluation method (such as cosine similarity, Jaccard similarity, etc.) may be used to further determine whether adjacent segmentation points have semantic overlap or duplication, so as to verify whether to perform the merging.

[0029] In step S140, the original dialogue text is segmented based on the target segmentation point set.

[0030] Here, the original dialogue text is segmented according to the target segmentation points using the merged target segmentation point set as a guide, thereby dividing the original dialogue text into multiple paragraphs. In some cases, each paragraph can be used to express a corresponding topic or context block.

[0031] Through the embodiment of the present application, the dual segmentation point extraction processing of the original dialogue text and the denoised text is combined, and the segmentation point extraction, merging and optimization strategies from multiple angles are adopted to achieve accurate segmentation of the dialogue text. By effectively removing noise information, the structural clarity of the text is enhanced, and through reasonable merging and segmentation operations, the robustness and accuracy of the segmentation are improved. It has strong adaptability and versatility, and can show excellent results in diversified dialogue scenarios.

[0032] It should be noted that in the segmentation of long dialogue texts, modeling more comprehensive semantic information is the key to the task. First, it is necessary to consider the fine-grained key information in the dialogue, and also to combine the corresponding context to focus on important information and identify complex main structures; second, in the process of modeling semantic information, it is necessary to exclude the interference of noise text to improve the model's adaptability to different types of dialogue texts and the robustness of segmentation.

[0033] At present, some relevant researchers have proposed to deal with the ambiguity of segment boundaries through neighborhood smoothing, so that the model can be more robust when dealing with segment boundaries and reduce the errors caused by boundary uncertainty; researchers have proposed a topic-aware sentence structure prediction task, which achieves data enhancement by exchanging topic paragraphs and randomly replacing text to help the model learn the structural information of sentence representation. In contrast, in the embodiment of the present application, a conference dialogue text segmentation method based on semantic complementarity is proposed, which can eliminate the noise interference inside the text and make the model pay more attention to the key information related to the dialogue topic, effectively improving the effectiveness of dialogue modeling and segmentation.

[0034] Figure 2 A schematic diagram of the architecture of an example of a text segmentation model according to an embodiment of the present application is shown.

[0035] like Figure 2 As shown in the figure, a new text segmentation model is proposed to address the particularity of conversation text or conference text. The model mainly consists of three modules: a conversation text denoising and topic-related sentence extraction module based on LDA, a conversation text segmentation point prediction module, and a segmentation point merging module based on similarity and density measurement. Specifically, the conversation text first passes through the first module to obtain the denoised text. The text before and after denoising is simultaneously sent to the conversation text segmentation prediction module. After obtaining two segmentation point sets respectively, they are input into the last merging module, and finally the overall segmentation position set is output.

[0036] For example, for the dialogue text segmentation task, the input is n Dialogue text sentences , after dividing the text into consecutive paragraphs, output the sequence number of the last sentence of each paragraph ,in .

[0037] Figure 3 An operational flowchart of an example of performing noise reduction processing on original conversation text according to an embodiment of the present application is shown.

[0038] like Figure 3 As shown, in step S310, at least one topic word corresponding to the original conversation text is determined based on the LDA model.

[0039] Here, LDA is a topic modeling algorithm based on text mining. The LDA model is used to perform topic modeling on the original conversation text. By learning the probability distribution of each word in the text, potential topics related to the conversation content are automatically extracted to obtain the corresponding topic word distribution.

[0040] In step S320, at least one topic sentence is extracted from the original conversation text, where the topic sentence is a sentence containing a topic word.

[0041] In some implementations, text matching techniques (such as TF-IDF, cosine similarity, etc.) can be used to find sentences containing these topic words in the original conversation text. These sentences are usually the core information in the conversation and represent the main content of the conversation. Since some sentences may contain redundant or incomplete information, such sentences can be filtered out during the extraction process, and repeated sentences in the conversation with low relevance to the topic can be eliminated.

[0042] In step S330, clustering calculation is performed on the original dialogue text based on each topic sentence, and clusters related to the topic are screened out to obtain the corresponding denoised dialogue text.

[0043] Here, based on the extracted topic sentences, various clustering algorithms (such as K-means, DBSCAN or hierarchical clustering) can be used to perform clustering calculations on the original conversation text, and the sentences can be divided into several clusters according to the semantic similarity between the sentences. Then, the clusters that are highly related to the topic are retained, and finally the denoised conversation text is obtained.

[0044] Through the embodiments of the present application, based on the LDA model and the subject word matching method, sentences related to the topic of the conversation can be accurately extracted. The text is effectively denoised by clustering to ensure that the output text is closely related to the topic, thereby improving the text quality and information density. Therefore, through topic modeling and cluster analysis, using the LDA-based conversation text denoising method, the conversation text containing effective information can be quickly extracted, and the denoised conversation text with a hierarchical topic structure of the conversation content can be realized.

[0045] It should be noted that some relevant studies have shown that important information in dialogue texts is reflected in high-frequency keywords, so the screening of noise sentences and invalid sentences in dialogue texts can be achieved by extracting high-frequency keywords that meet the theme of the segment. For keyword extraction, topic modeling technology can be used, which can discover potential topics in a document collection. Specifically, LDA assumes that each document is composed of multiple topics, and each topic is defined by a vocabulary distribution, that is, it models the document-topic distribution and topic-word distribution. The goal of LDA is to reverse the topic distribution of each document and the word distribution of each topic by observing the word distribution in the document. LDA is a bag of words model, that is, it believes that a document is a set composed of a group of words, and there is no order or precedence between words. The word frequency is an implicit consideration of LDA, which directly affects LDA's estimation of the topic distribution. Therefore, the noun subset in the topic word distribution after LDA modeling can be used as the high-frequency keywords required by the embodiment of the present application.

[0046] After obtaining the topic words, all sentences containing the topic words in the conversation are extracted and clustered based on their Euclidean distance in the original conversation. The purpose is to filter out the density-connected clusters in the conversation text so that the extracted conversation text is more semantically fluent, denoted as ,in ,in: and .

[0047] Figure 4 A schematic diagram showing the comparative effect of an example of a sentence before and after noise removal is shown.

[0048] like Figure 4 As shown in the figure, in the original text, we can see a lot of colloquial expressions and filler sentences that are irrelevant to the core topic, such as "Yes", "Yes, um, ah. Um.", etc. Although these sentences are common in conversations, they are not meaningful in expressing the core content or promoting the topic of the conversation, and are typical noise content. In the denoised text, these contents are effectively deleted, and the denoised text retains the core content of the conversation, making the overall semantics more focused and clear.

[0049] Figure 5 An operation flow chart of an example of extracting a segmentation point set according to an embodiment of the present application is shown.

[0050] like Figure 5 As shown, in step S510, the original dialogue text and the noise-removed dialogue text are divided into a first sentence set and a second sentence set, respectively.

[0051] Here, the original dialogue text and the denoised dialogue text are segmented at the sentence level, and various non-restrictive sentence segmentation methods can be used, such as segmentation based on punctuation marks (such as periods, question marks, and exclamation marks, etc.), or auxiliary sentence segmentation in conjunction with word segmentation tools, which are not limited here. In some examples, the same sentence segmentation technology can be applied to the original text and the denoised text, respectively, to ensure that the number of sentences segmented by the two is consistent and the positions are corresponding.

[0052] In step S520, based on the end sentence classification model, at least one first paragraph end sentence in the first sentence set and at least one second paragraph end sentence in the second sentence set are identified; the paragraph end sentence is the last sentence of the paragraph.

[0053] Here, the end sentence classification model can use various supervised learning models to identify whether each sentence is a paragraph end sentence. For example, the end sentence usually contains summary, ending or transitional language (such as "so that's it", "in general", etc.), and the end sentence may be shorter than an ordinary sentence, or contain obvious pauses and punctuation marks, etc. By learning the characteristic patterns of such paragraph end sentences through sample supervision, the identification of paragraph end sentences can be effectively realized.

[0054] In step S530, a corresponding first segmentation point set and a corresponding second segmentation point set are constructed according to the position of each last sentence of the first paragraph and the position of each last sentence of the second paragraph.

[0055] Exemplarily, for each final sentence, the position index of the final sentence is recorded according to its position in the sentence set. For example, if the final sentence is the fifth sentence in the sentence set, its position index is 5. In addition, the final sentence position index extracted from the first sentence set is used to construct a first segmentation point set, which can reflect the paragraph structure of the original dialogue text. The final sentence position index extracted from the second sentence set is used to construct a second segmentation point set, which is based on the segmentation points extracted from the content after noise reduction processing, and the paragraph structure will be more refined and can highlight the key points.

[0056] Through the embodiments of the present application, the segmentation point extraction method based on the last sentence classification model is combined with the original dialogue text and the denoised dialogue text. By identifying the last sentence of each paragraph, the dialogue text can be accurately segmented, avoiding the problem of excessive or insufficient segmentation.

[0057] In some examples of the embodiments of the present application, the end-sentence classification model adopts a word-gram classification model and is used to identify whether the first word-gram and / or the last word-gram of each sentence in a sentence set is a paragraph-ending word-gram, so as to determine the sentence containing the paragraph-ending word-gram as the paragraph-ending sentence.

[0058] It should be noted that a word-unit is a basic concept in natural language processing. It is the smallest semantic unit in the text processing process. For example, a word-unit can be a word, a punctuation mark, or a special symbol (such as a number, date, etc.).

[0059] As described above, in a conversation or meeting, the first or last word of the ending sentence of a paragraph of a conversation or meeting is usually marked by a specific landmark word (such as "summary" words, modal particles, pause symbols, etc.) or grammatical structure (such as certain sentence patterns or question words, etc.), for example, "next discussion", "in short", "in summary", "thank you", "OK, I understand", etc. Therefore, a word-unit classification model is designed to learn such feature patterns to achieve the recognition of the ending sentence of a paragraph.

[0060] It should be noted that in order to train the end-of-sentence classification model, we first need to label each sentence in the conversation and determine which first and last words of the sentences are the end signs of the paragraph. When labeling, the characteristics of the end-of-paragraph words in different contexts should be considered.

[0061] More specifically, for a given dialogue text or , use the pre-trained model to encode each sentence and make a judgment. The judgment condition is whether it is the last sentence of the paragraph. The set of sequence numbers of sentences judged as "yes" is the predicted segmentation T. In detail, the dialogue segmentation task is trained and inferred as a word classification task. For the conference dialogue text , divided into sentences according to the granularity of discourse, we get , add a separator at the end of each sentence <eop>,get , and model the entire text: , formula (1) Then, use the word classification model to classify each <eop>The word unit is judged as binary classification. The word unit classification model can include a fully connected layer and a classification layer (which can use a softmax activation function). If the judgment is "yes", the <eop>The sentence where the word is located is regarded as the last sentence of the paragraph. Thus, the segmentation of the entire text is obtained: , formula (2) in, for The feature vector corresponding to the EOP word in, ; p Predict the probability distribution for the segmentation of all EOP tokens. During training, for the label sequence , the model minimizes the cross entropy loss of segmentation prediction to optimize the model weights: , formula (3) In some examples of the embodiments of the present application, joint training of semantically complementary texts can also be implemented for the word-meta classification model. The tag sequence and the predicted probability distribution , which can simultaneously minimize the sum of the cross entropy of the two texts, for The closest (including itself) segmentation position in the denoised text sequence: , formula (4) In the formula, represents the loss function, n Indicates the number of sentences in the dialogue text, Represents the index of the sentence; Indicates the first The label of the word of the sentence, Indicates that the model predicts the first The probability that the word of a sentence is the end word of a paragraph, Indicates that the model predicts the first The probability that the token of a sentence is not the end token of a paragraph; Indicates the first The label of the word of the sentence, The model predicts the first The probability that the word of a sentence is the end word of a paragraph, The model predicts the first The probability that the token of a sentence is not the end-of-paragraph token.

[0062] Through the embodiment of the present application, the loss function comprehensively considers the prediction of the end-of-paragraph word in the original dialogue text and the denoised dialogue text, and the model needs to minimize the cross-entropy loss in both texts at the same time. Through this joint training method, by jointly modeling texts at two different information levels, the model can learn complete contextual features from the original text, while strengthening the ability to recognize topic-related features from the denoised text.

[0063] Figure 6 An operational flow chart of an example of merging a set of segmentation points according to an embodiment of the present application is shown.

[0064] like Figure 6 As shown, in step S610, the second segmentation point set is mapped to the original dialogue text to determine a corresponding mapping segmentation point set.

[0065] It should be noted that the second segmentation point set is extracted based on the denoised dialogue text. Since some noise content may be deleted during the denoising process, the position of the segmentation point in the denoised text may not correspond to the actual position in the original text. In order to map the second segmentation point set back to the original dialogue text, a mapping rule needs to be defined, which can be a mapping based on sentence index (e.g., position mapping) or a mapping based on content similarity (e.g., semantic association), etc.

[0066] In step S620, the first segmentation point set and the mapping segmentation point set are merged to obtain a corresponding target segmentation point set.

[0067] For example, if the segmentation points in the two sets are very close in position (for example, the index difference is within a certain range, such as 1-2 sentences), they can be merged into one segmentation point, and the segmentation point with higher confidence is preferentially retained. In addition, if the segmentation points in the two sets are far apart in position, the mapped segmentation points can be directly added to the first segmentation point set to achieve clearer paragraph segmentation of the original dialogue text.

[0068] The following will use examples to expand the details of the segmentation point merging module based on similarity, NSP (Next Sentence Prediction, sentence pair matching) and density measurement.

[0069] After getting the segmentation prediction probability distribution and After that, you can use the argmax function to get the segmentation prediction and As for the two segmentation results based on texts with different semantic connotations, it is believed that they take into account the coherent semantics and information semantics of the dialogue text respectively. Therefore, merging these two segmentation results has obtained a more comprehensive segmentation result. In the merging process, the segmentation point set of the denoised text is firstly Mapping onto the original sequence yields , and Based on An additional segmentation is inserted into it.

[0070] In some examples of the embodiments of the present application, for each first segmentation point in the first segmentation point set, the similarity distance between the context information corresponding to each mapping segmentation point in the mapping segmentation point set and the context information corresponding to the first segmentation point is calculated, and the mapping segmentation points whose corresponding similarity distance is less than a preset retrieval distance threshold are screened out. Then, the screened mapping segmentation point set and the first segmentation point set are merged to obtain a corresponding target segmentation point set.

[0071] Specifically, the NSP task is one of the pre-training tasks of BERT, which aims to determine whether sentence B is the next sentence of sentence A in the given two sentences (Sentence A and Sentence B). In this embodiment, this task is applied to determine whether the segmentation position is correct. Each additional split point in ,in , take it to The text between adjacent split points is used as the split point Since too long a context is not helpful for similarity judgment, when the context is too long, the adjacent text can be cut according to a fixed length.

[0072] For each split point , by using the pre-trained language model to model its corresponding context and following the NSP task process to predict whether the context is coherent: , formula (5) , formula (6) Then, high-weight segmentation points are removed to ensure that coherent dialogue text is not segmented.

[0073] Figure 7 A schematic diagram showing the effect of an example of segmentation point density merging based on a clustering algorithm according to an embodiment of the present application is shown.

[0074] like Figure 7 As shown in the figure, due to the missing of some text in the denoised dialogue text, some of its segmentation points will be slightly offset relative to the original text after mapping, which does not affect the semantic understanding of the text. and The density metric between is used to merge such adjacent segmentation points, which solves the problem of the offset of the mapping segmentation points in the original text caused by the missing of some content in the denoised dialogue text. Figure 7 In it, although there are offset segmentation points in the two parts of the text "dui" and "shi de", based on their positions and semantic density metrics, they can be merged into one segmentation point. Based on the merged segmentation point, the semantic boundary in the original text can be correctly reflected, ensuring the logical consistency of segmentation.

[0075] Regarding the details of this clustering algorithm, the set of segmentation points of the original text The points in are regarded as core points, and a retrieval distance is set. For each core point, The segmentation points in with an Euclidean distance less than the retrieval distance are boundary points, and the rest are noise points. After the retrieval is completed, for all boundary points, they can be merged into the corresponding core points, that is, these boundary points are removed. For the remaining noise points, density merging can also be performed to ensure that adjacent segmentation points that are too close are not inserted.

[0076] Figure 8 The experimental simulation diagram showing an example of the influence of different retrieval distances on the final segmentation effect according to the embodiments of the present application is shown.

[0077] As Figure 8 shown, for the hyperparameters of segmentation point merging, they are calculated by performing calculations on the validation set. The experiment evaluates the merging results of segmentation points on the validation set, and uses four metrics (1-pk, 1-wd, bs, a) to quantitatively evaluate the final segmentation effect, and compares the segmentation performance of the model under different parameters. is a key parameter of the density clustering algorithm (such as DBSCAN), which is used to define the retrieval radius during clustering, that is, the distance threshold for determining whether a segmentation point belongs to the same cluster. During the segmentation point merging process, determines which adjacent segmentation points will be clustered into the same segmentation point cluster and finally merged into a single target segmentation point.

[0078] Specifically, 1-pk (red line) is used to measure the accuracy of segmentation point prediction, and the higher the score, the more accurate the segmentation. 1-wd (green line) is used to measure the consistency of the distribution of segmentation points between paragraphs, and the higher the score, the more reasonable the position distribution of the segmentation points. bs (blue line) represents the suitability of the segmentation granularity after the segmentation points are merged, and the higher the value, the more in line with the expectation the paragraph segmentation granularity is. a (yellow line) represents the comprehensive score of the overall segmentation performance.

[0079] The experimental results show that when the model performs best in all metrics, indicating that this retrieval distance can achieve a good balance between removing redundant segmentation points and maintaining the actual semantic boundary. In addition, a smaller (such as 15-20) will cause some segmentation points that should be merged to not be clustered, thus affecting the segmentation accuracy and granularity. (such as 30-35) will cause some segmentation points to be over-merged, destroying the semantic boundaries of the actual paragraphs and reducing the accuracy of segmentation point prediction.

[0080] In the process of practicing the present application, the inventor of the present application also conducted experiments to demonstrate the relevant effects of the technical solution of the present application. The corresponding experimental details will be specifically expanded below.

[0081] In the embodiment of the present application, the final experiment was implemented based on the PyTorch library. Throughout the experiment, TopicSegModel (a language model pre-trained with business data), Roberta and Bert were used as the benchmark models for segmentation, calculation of NSP probability and detection of entities. TopicSegModel is a model obtained by adding business-related data for additional training on the basis of the existing pre-trained model. For comparison, some baseline methods and current advanced models and methods were selected for comparison; at the same time, in order to explore the development of LLMs in the field of long dialogue text segmentation, the current mainstream online large-scale language models were also compared. Among them, TopicSegModel is a benchmark model for MUG tasks, and its effect is good, so the relevant work of the embodiment of the present application is experimentally analyzed based on TopicSegModel.

[0082] Adams et al. used entity density to evaluate text quality. Here, a pre-trained Bert-ner (sequence labeling) model is also used on the AMC dataset to extract entities from the conversation text and calculate the number to evaluate the denoised conversation text.

[0083] Table 1. Analysis of AMC dataset As shown in Table 1, the entity density in the extracted denoised dialogue text is much higher than that in the original dialogue text. D 'Compared to D There is less noise and higher information density in the original text. The original text has more comprehensive and coherent semantic information, which does not need to be proved. The semantics of the two are complementary, which has been proven to be feasible.

[0084] Table 2. Comparative experimental results As shown in Table 2, in the comparative experiment, a variety of advanced methods were selected for comparison. As shown in the table, it can be seen that the method provided in the embodiment of the present application is the best in all indicators (A, B, S, 1-WD, 1-Pk are all internationally common segmented evaluation indicators).

[0085] Among them, for Transformer2BERT, only the classification model is trained on the segmentation task; since the sentence embedding used by the hierarchical model will lose a lot of semantic information compared to the full-text features, the effect is general. The topic-aware sentence structure prediction method is not effective. It can be inferred that this method improves the robustness of the model by adding noise to the data. It is feasible to use it in serious text datasets such as WikiSection, but the dialogue text itself contains a lot of noise, which makes TSSP unable to improve the model performance in this case. For the method provided in the embodiment of the present application, it is better to train the two data with separate models than to train them together. This is because the understanding of the two semantics will cause conflicts for the same model. Using two models to train separately can better focus on specific semantics, so as to obtain better segmentation after the fusion results. Large-scale language models (Large Language Models, LLMs) such as GPT-4 have made amazing achievements in language understanding, especially for the understanding of ultra-long texts, which far exceeds the existing small models. In this regard, several mainstream online large-scale language models are compared. We selected part of the test set (the first 65 items) for comparison, used appropriate prompts as zero-shots (no examples), and ensured that the model outputs stable results. The segmentation results are shown in Table 3 below.

[0086] Table 3. Segmentation results It can be seen that LLMs still achieves good results in the long text understanding task even in zero-shot, but it is still slightly inferior to the method provided in the embodiment of the present application.

[0087] Through the embodiment of the present application, a long conference dialogue text segmentation method based on semantic complementarity is proposed. The method first filters the original dialogue text to generate low-noise dialogue text, and uses a pre-trained language model to simultaneously model and segment the original dialogue text with coherent semantics and the low-noise text with high-density information semantics, and merges the segmentation points according to the coherence between paragraph texts and the density of segmentation points, thereby integrating the semantic advantages of both to obtain better segmentation results. Compared with general segmentation models, the method provided in the embodiment of the present application takes into account the semantic coherence of the dialogue text and the distribution of key dialogue information at the same time, and has obvious improvements on relevant data sets.

[0088] Specifically, a conversation text denoising method based on LDA is proposed to quickly extract conversation texts containing effective information. The topic segmentation of long conference conversation texts is enhanced by achieving semantic complementarity of conversation texts. This method is model-independent and applicable to various topic meetings and language models. In addition, it also pays attention to the coherence of the conversation text and the effective key information contained in the conversation, achieving a more comprehensive semantic modeling of the conversation text.

[0089] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0090] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned conversation text segmentation methods of the present application.

[0091] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any one of the above-mentioned conversation text segmentation methods.

[0092] In some embodiments, an embodiment of the present application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the conversation text segmentation method.

[0093] Fig. 9 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for executing a method for segmenting a conversation text provided by another embodiment of the present application. Fig. 9 As shown, the device includes: One or more processors 910 and memory 920, Fig. 9 A processor 910 is taken as an example.

[0094] The device for executing the conversation text segmentation method may further include: an input device 930 and an output device 940 .

[0095] The processor 910, the memory 920, the input device 930 and the output device 940 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0096] The memory 920 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the conversation text segmentation method in the embodiment of the present application. The processor 910 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 920, that is, the conversation text segmentation method in the above method embodiment is implemented.

[0097] The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 920 may optionally include a memory remotely arranged relative to the processor 910, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The input device 930 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 940 may include a display device such as a display screen.

[0099] The one or more modules are stored in the memory 920, and when executed by the one or more processors 910, the conversation text segmentation method in any of the above method embodiments is executed.

[0100] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0101] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0102] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0103] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0104] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.< / eop> < / eop> < / eop>

Claims

1. A method for segmenting a conversation text, comprising: Performing noise reduction processing on the original dialogue text to obtain the corresponding noise-removed dialogue text; Extracting a corresponding first segmentation point set and a second segmentation point set from the original dialogue text and the denoised dialogue text respectively; Merging the first segmentation point set and the second segmentation point set to obtain a target segmentation point set; The original dialogue text is segmented based on the target segmentation point set.

2. The method according to claim 1, wherein: The denoising process is performed on the original dialogue text to obtain the corresponding denoised dialogue text, including: Determine at least one topic word corresponding to the original conversation text based on the LDA model; Extracting at least one topic sentence from the original conversation text, wherein the topic sentence is a sentence containing the topic word; The original dialogue text is clustered based on each of the topic sentences, and clusters related to the topic are screened out to obtain the corresponding denoised dialogue text.

3. The method according to claim 1, wherein: The extracting the corresponding first segmentation point set and the second segmentation point set from the original dialogue text and the denoised dialogue text respectively comprises: The original dialogue text and the denoised dialogue text are divided into a first sentence set and a second sentence set respectively; Based on the end sentence classification model, identifying at least one first paragraph end sentence in the first sentence set and at least one second paragraph end sentence in the second sentence set; the paragraph end sentence is the last sentence of the paragraph; According to the position of each of the last sentences of the first paragraphs and the position of each of the last sentences of the second paragraphs, a corresponding first segmentation point set and a corresponding second segmentation point set are respectively constructed.

4. The method according to claim 3, wherein: The end-of-sentence classification model adopts a word-unit classification model and is used to identify whether the first word-unit and / or the last word-unit of each sentence in the sentence set is a paragraph-end word-unit, so as to determine the sentence containing the paragraph-end word-unit as the paragraph-end sentence.

5. The method according to claim 4, wherein: The loss function of the last sentence classification model is: In the formula, represents the loss function, n Indicates the number of sentences in the dialogue text, Represents the index of the sentence; Indicates the first The label of the word of the sentence, Indicates that the model predicts the first The probability that the word of a sentence is the end word of a paragraph, Indicates that the model predicts the first The probability that the token of a sentence is not the end token of a paragraph; Indicates the first The label of the word of the sentence, The model predicts the first The probability that the word of a sentence is the end word of a paragraph, The model predicts the first The probability that the token of a sentence is not the end-of-paragraph token.

6. The method according to claim 1, wherein: The merging the first segmentation point set and the second segmentation point set to obtain a target segmentation point set includes: Mapping the second segmentation point set to the original dialogue text to determine a corresponding mapping segmentation point set; The first segmentation point set and the mapping segmentation point set are merged to obtain a corresponding target segmentation point set.

7. The method according to claim 6, wherein: The merging the first segmentation point set and the mapping segmentation point set to obtain a corresponding target segmentation point set includes: For each first segmentation point in the first segmentation point set, calculating a similarity distance between context information corresponding to each mapping segmentation point in the mapping segmentation point set and context information corresponding to the first segmentation point, and filtering out mapping segmentation points whose corresponding similarity distances are smaller than a preset retrieval distance threshold; The filtered mapping segmentation point set and the first segmentation point set are fused to obtain a corresponding target segmentation point set.

8. A storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images