Text processing method and device, electronic device and computer-readable storage medium
By segmenting and topic clustering the text recorded in speech, and generating subtopic texts with sentence position information, the problem of inefficient topic organization in the existing technology is solved, and a more efficient text organization and search experience is achieved.
Patent Information
- Application Number
- CN202011156091.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-10-26
AI Technical Summary
In the prior art, the subject organization based on the original text recorded by voice is inefficient, which leads to inconvenience in searching and reviewing in the future.
By obtaining the original text, dividing paragraphs according to the similarity between sentences, and combining the position information of the sentences to cluster and segment the topics to generate sub-topic texts, so that users can organize and search according to different topics.
It improves the convenience of users when searching and reading, and enhances the efficiency and user experience of text organization.
Smart Images

Figure CN114492375B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of text processing, and in particular to a text processing method and device, an electronic device, and a computer-readable storage medium for processing original text. Background Art
[0002] With the development of society, people use text or voice to record various operations more and more frequently, such as students taking notes in class, employees recording the content of meetings, etc. In this case, the content people record is becoming more and more complicated, and therefore such long paragraphs of recorded text content bring great inconvenience in the future when sorting and searching. For example, in the case of taking notes in class, students can usually only record what the teacher says or writes on the blackboard as the class time goes by, which is usually reflected in the text as a diary-like record, but after class or at the end of the semester, it is necessary to search or study according to the corresponding syllabus of this class. Therefore, it is difficult to find the content you need from a long diary-like record. Summary of the invention
[0003] Embodiments of the present application provide a text processing method and apparatus, an electronic device, and a computer-readable storage medium to address the defect of low efficiency in subject sorting of original texts recorded based on voice in the prior art.
[0004] To achieve the above object, the present application provides a text processing method, including:
[0005] Get the original text containing multiple sentences;
[0006] Dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original text to generate segmented text;
[0007] The generated segmented text is clustered based on the position information of each sentence in the segmented text, and the segmented text is segmented based on the topic clustering result to generate sub-topic text.
[0008] The present application also provides a text processing device, including:
[0009] A first acquisition module is used to acquire an original text containing a plurality of sentences;
[0010] A first segmentation module, configured to divide the plurality of sentences into at least two paragraphs according to the similarity between the sentences in the original text, so as to generate segmented text;
[0011] The first topic segmentation module is used to perform topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and perform topic segmentation processing on the segmented text according to the topic clustering result to generate sub-topic text.
[0012] The present application also provides a text processing method, including:
[0013] Get the original classroom transcript containing multiple sentences;
[0014] Dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original classroom records to generate a segmented text of the classroom records;
[0015] The generated classroom record segmented text is topic clustered in combination with the position information of each sentence in the classroom record segmented text, and the classroom record segmented text is topic segmented according to the topic clustering result to generate sub-topic text.
[0016] The present application also provides a text processing method, including:
[0017] Get meeting transcripts containing multiple sentences;
[0018] Dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the meeting minutes to generate segmented text;
[0019] The generated segmented text is clustered based on the position information of each sentence in the segmented text, and the segmented text is segmented according to the topic clustering result to generate a sub-topic text.
[0020] The present application also provides an electronic device, including:
[0021] Memory, used to store programs;
[0022] The processor is used to run the program stored in the memory, and when the program is run, the text processing method provided in the embodiment of the present application is executed.
[0023] An embodiment of the present application also provides a computer-readable storage medium on which a computer program executable by a processor is stored, wherein the program, when executed by the processor, implements the text processing method provided in the embodiment of the present application.
[0024] The text processing method and device, electronic device and computer-readable storage medium provided in the embodiments of the present application can consider the position of sentences in the original text when segmenting the original text, so as to divide sentences that are close in time sequence into one segment as much as possible to ensure the coherence of the sentences in terms of topic, and then perform topic clustering on the generated segmented text, and perform topic segmentation on the text according to the topic clustering results to generate topic-divided text, so as to automatically provide users with text that has been segmented and organized according to different topics, which facilitates users' future search and reading and improves user experience.
[0025] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0027] Figure 1 A schematic diagram of an application scenario of the text processing method provided in an embodiment of the present application;
[0028] Figure 2 A flowchart of an embodiment of a text processing method provided by the present application;
[0029] Figure 3 A flowchart of another embodiment of the text processing method provided by the present application;
[0030] Figure 4 A schematic diagram of the structure of an embodiment of a text processing device provided by the present application;
[0031] Figure 5 A schematic diagram of the structure of an electronic device embodiment provided in this application. DETAILED DESCRIPTION
[0032] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0033] Embodiment 1
[0034] The solution provided in the embodiment of the present application can be applied to any system with text processing capabilities. Figure 1 A schematic diagram of an application scenario of the text processing method provided in an embodiment of the present application, Figure 1 The scenario shown is only one example of a scenario to which the technical solution of the present application can be applied.
[0035] In daily life and work, people often need to use text to record the voices heard in daily life or especially at work, in order to form record archives or facilitate future inquiries and references. However, with the development of science and technology and society, the content that people need to record at one time is becoming more and more and more complex. Therefore, in such cases, it is often necessary to use very long text records to record the entire voice content. For example, students use text to record the teacher’s oral teaching content in class and / or the teacher’s writing on the blackboard. However, due to the long length of such recorded diary-like text, students need to spend a long time to find the desired content when reviewing in the future, or need to spend time after the class to organize the recorded content according to the teaching content of this lesson, such as the syllabus. For example, Figure 1 As shown in the scene where students record the content of the class, the teacher said the following in class: "Okay, let's start the class. This. From today on, we are in the next semester. The main content is the integration of multivariate functions. Today we will talk about the chapter of triple integrals. So in this chapter, it is mainly about the definition, calculation method and application." Usually, students can use, for example, a voice recorder to record the teacher's voice in the form of audio, and can convert the audio into the above text through various voice-to-text software. Or in the case of distance education, the system server can automatically convert the above voice narrated by the teacher into text and send it to each student as class notes. In the prior art, students can only save the diary-style text record as the teaching notes of the class, but the teaching notes saved in this way are very inconvenient to find and refer to when students review in the future. Or students can also manually organize the text record by themselves. For example, students can manually organize the text record according to the syllabus corresponding to the class provided by the teacher in advance to the students or according to the teaching content corresponding to the class in the textbook. Such organization work is also very time-consuming and laborious for students.
[0036] Therefore, if Figure 1 As shown in the embodiment of the present application, the segmented text can be generated by automatically segmenting the original text. Therefore, after the original text is segmented, the following is obtained: Figure 1The three segmented texts 1-3 shown in the figure are segmented text 1 "Okay, let's start the class", segmented text 2 "This. From today on, we are in the next semester. The main content is the integration of multivariate functions" and segmented text 3 "Today we will talk about the chapter on triple integrals. So in this chapter, it is mainly about the definition, calculation method and application". After that, segmented texts 1-3 can be processed by topic clustering. For example, Figure 1 As shown in , the segmented texts 1-3 can be topic clustered according to the similarity between the sentences. In particular, in the similarity calculation, the embodiment of the present application introduces the position information of each sentence in the text to ensure that sentences with similar positions in the original text can have a higher similarity, thereby ensuring the linguistic coherence and order of the recorded sentences.
[0037] For example, Figure 1 As shown in , we can first calculate the similarity between sentences in the segmented text. However, since the recorded classroom speech text is usually more colloquial, it is very likely that the meanings of two adjacent sentences are not similar, but they actually involve the same content. For example, in Figure 1 In the scene shown in , segmented text 2 contains two sentences, "From today on, we are in the next semester. The main content is the integration of multivariate functions." From the meaning of these two sentences, the first sentence is related to the current time, that is, today is the next semester, and the second sentence is related to the teaching content, that is, the content of the next semester is the integration of multivariate functions. Therefore, if only from the meaning of this sentence, the similarity of these two sentences is low, but from the overall point of view, these two sentences should have a strong correlation. Therefore, in an embodiment of the present application, by introducing the position information of the sentences in the text, it is possible to make such two sentences with different meanings have a higher similarity, thereby ensuring that the two sentences are identified as being for the same topic. Afterwards, segments 1 and 2 can be determined as topic 1 according to the results of clustering, and segment 3 can be determined as topic 2. The topic strength of each segmented text 1-3 can be calculated according to a predetermined number of categories.
[0038] For example, Figure 1 As shown in , the topic strength values of segmented texts 1-3 in two categories can be calculated, and then the difference between the topic strength values can be calculated. The sentences with the difference less than the predetermined threshold are used as transition sentences between the corresponding category topics, and the sub-topic texts are generated according to the transition sentences. For example, Figure 1As shown in , the topic intensity values on topic 1 in segment 1 and segment 2 are relatively high, and the topic intensity values on topic 2 are correspondingly relatively low, while segment 3, because it involves content that is obviously different from segment 1 and segment 2, has a relatively low topic intensity value on topic 1 and a relatively high topic intensity value on topic 2. Specifically, the topic intensity values on topic 1 and topic 2, i.e., topic relevance, can be calculated for each sentence in segments 1-3. And therefore, for example, the topic intensity values on topics 1 and 2 of the sentence "We will talk about the chapter on triple integrals today" in segment 3 are relatively close to the topic intensity values on topics 1 and 2 of the last sentence in segment 2 "The main content is the integration of multivariate functions". Therefore, in the embodiment of the present application, the topic intensity values on each topic of each sentence can be calculated by difference, and the sentence with a smaller difference in topic intensity values on two different topics can be regarded as having relevance to both topics, and therefore, it is likely to be a transition sentence between the two topics. Therefore, topic segmentation can be performed based on the transition sentences determined in this way, so that segments 1 and 2 are divided into topic 1, and segment 3 is divided into topic 2 using the last sentence of segment 2 and the first sentence of segment 3 as segmentation points.
[0039] Therefore, by adopting the text processing scheme of the embodiment of the present application, the original text can be segmented and then the position of the sentences in the segmented text can be considered when topic clustering is performed on the generated segmented text, so that sentences that are close in time sequence have a high similarity to ensure the coherence of the sentences in terms of topic, and the text is topic-segmented according to the topic clustering results to generate topic-segmented texts, thereby automatically providing users with texts that have been segmented and organized according to different topics, facilitating users' future searches and readings and improving user experience.
[0040] The above embodiments are illustrations of the technical principles and exemplary application frameworks of the embodiments of the present application. The specific technical solutions of the embodiments of the present application are further described in detail below through multiple embodiments.
[0041] Embodiment 2
[0042] Figure 2 This is a flowchart of an embodiment of the text processing method provided by the present application. The execution subject of the method can be various terminals or server devices with text processing capabilities, or it can be a device or chip integrated on these devices. Figure 2 As shown, the text processing method includes the following steps:
[0043] S201, obtaining an original text containing a plurality of sentences.
[0044] In an embodiment of the present application, an original text containing multiple sentences can be obtained based on an audio file of a target user's voice recording. For example, during a meeting, a device with an audio recording function such as a recorder can be used to record the speech of a participant, and then it can be converted into a text file through various speech recognition algorithms, thereby generating an original text corresponding to the speech of the participant of the meeting. In addition, in an embodiment of the present application, the recorded voice can be converted into corresponding text in real time in a scene such as a meeting through automatic speech recognition (ASR) technology, so that the original text corresponding to the speech of the participant of the meeting can be directly generated at the end of the meeting.
[0045] S202, dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original text, so as to generate segmented text.
[0046] After the original text containing multiple sentences is obtained in step S201, since the original text obtained in step S201 is generated based on the speech record of the speaker in the scene of a meeting or classroom, such original text often has the form of a running account, especially containing various spoken words and various transition words or even sentences that connect the previous and the next. Therefore, the granularity of each sentence recorded in this way is also small, and the amount of information contained is correspondingly small. Therefore, in step S202, such a running account-style original text obtained in step S201 can be first divided into paragraphs. For example, in step 202, the similarity between each sentence in the original text can be calculated first, and the paragraph division process can be performed based on the calculated similarity.
[0047] S203, performing topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and performing topic segmentation processing on the segmented text according to the topic clustering result to generate topic-divided text.
[0048] After obtaining the segmented text in step S202, topic clustering can be performed on the obtained segmented text. For example, various topic clustering algorithms such as LDA (Latent Dirichlet Allocation) can be used to perform clustering calculations. In particular, in the embodiment of the present application, since the original text recorded in step S201 is usually more colloquial, it is likely that two consecutive words spoken in time sequence will be considered unrelated when topic clustering is performed separately. For example, Figure 1As shown in , segmented text 2 contains two sentences, "From today on, we are in the next semester. The main content is the integration of multivariate functions." From the meaning of these two sentences, the first sentence is related to the current time, that is, today is the next semester, and the second sentence is related to the teaching content, that is, the content of the next semester is the integration of multivariate functions. Therefore, if we only look at the meaning of this sentence, the similarity of these two sentences is low, but from the overall point of view, these two sentences should have a strong correlation. Therefore, in an embodiment of the present application, by introducing the position information of the sentences in the text, it is possible to make such two sentences with different meanings have a higher similarity, thereby ensuring that the two sentences are identified as being for the same topic. Therefore, in step S203, the position of each sentence in the segmented text can be considered when performing topic clustering, so that adjacent sentences can have a higher correlation, thereby ensuring the coherence and sequentiality of the topic.
[0049] Therefore, by utilizing the text processing method of the embodiment of the present application, the original text can be segmented and then the positions of the sentences in the segmented text can be considered when topic clustering is performed on the generated segmented text, so that sentences that are close in time sequence have a high similarity to ensure the coherence of the sentences in terms of topic, and the text is topic segmented according to the topic clustering results to generate topic-based texts, thereby automatically providing users with texts that have been segmented and organized according to different topics, facilitating users' future searches and readings and improving user experience.
[0050] Embodiment 3
[0051] Figure 3 This is a flowchart of another embodiment of the text processing method provided by the present application. The execution subject of the method can be various terminals or server devices with text processing capabilities, or it can be a device or chip integrated on these devices. Figure 3 As shown, the text processing method includes the following steps:
[0052] S301, obtaining an original text containing a plurality of sentences.
[0053] In an embodiment of the present application, an original text containing multiple sentences can be obtained based on an audio file of a target user's voice recording. For example, during a meeting, a device with an audio recording function such as a recorder can be used to record the speech of a participant, and then it can be converted into a text file through various speech recognition algorithms, thereby generating an original text corresponding to the speech of the participant of the meeting. In addition, in an embodiment of the present application, the recorded voice can be converted into corresponding text in real time in a scene such as a meeting through automatic speech recognition (ASR) technology, so that the original text corresponding to the speech of the participant of the meeting can be directly generated at the end of the meeting.
[0054] S302, calculating the number of words in each sentence in the original text.
[0055] S303: Merge the sentence whose word count is less than a predetermined word count threshold with the sentence adjacent to the sentence in position, until the number of valid words meeting a predetermined standard in the merged sentence is greater than a predetermined valid word threshold.
[0056] In the embodiment of the present application, after the original text is obtained in step S301, since the original text is usually generated based on the speech record of the speaker in the scene such as a meeting or classroom, it is usually more colloquial in form, especially each sentence may be relatively short. Figure 1 As shown in , the original text may include the two sentences "This. From today on, we are in the next semester", and the first sentence actually contains only two words "this". Therefore, if paragraph division and subsequent topic segmentation are performed based on such sentences, the granularity will be too small due to too few words in the sentences, and the amount of information will be too small, making it difficult to process as a sentence unit. Therefore, in an embodiment of the present application, the number of words in each sentence in the original text obtained in step S301 can be calculated in step S302, and in step S303, the sentence whose number of words calculated in step S302 is less than a predetermined word count threshold, for example, 5, is regarded as a sentence with too small granularity. For example, as described herein, Figure 1The sentence "this" contained in the original text shown contains only two characters, so it is obviously a sentence with too fine a granularity. Therefore, in step S303, this sentence can be merged with the sentence adjacent to it in position, that is, merged with the subsequent sentence "From today on, we are in the next semester" into one sentence. Thus, the number of characters contained in the merged sentence is obviously greater than 5, so it can be used as a sentence unit for subsequent paragraph segmentation and topic division. In addition, in the embodiments of the present application, the number of valid words that meet the predetermined criteria in the merged sentence can also be further counted. For example, in the sentence "From today on, we are in the next semester" mentioned above, words other than modal particles such as "呢", "了", and "啊" can be regarded as valid words, and the number of such valid words can be counted to determine whether further merging with adjacent sentences is needed. Therefore, through such processing, it can be ensured that the sentence units for subsequent paragraph segmentation and topic processing contain enough meaningful words, which can further ensure the accuracy and efficiency of the calculation. In addition, in the embodiments of the present application, whether a word is a stop word can also be used to determine valid words, that is, each sentence in the original text is screened according to a predetermined list or database of stop words, so as to take the words other than the stop words as valid words.
[0057] S304. Calculate the difference between the peak similarities of each sentence and at least two sentences adjacent to it in position in the original text as the relative similarity of this sentence.
[0058] S305. Determine the paragraph division points for paragraphing the original text according to the relative similarities.
[0059] S306. Generate segmented text based on the paragraph division points.
[0060] After the original text containing multiple sentences is obtained in step S301, or after the sentences are merged in steps S302 and S303, the similarity between adjacent sentences in the segmented text can be calculated in step S304. For example, in the present application, the cosine value can be used to calculate the similarity between adjacent sentences. In addition, in the embodiment of the present application, some pre-trained word vectors can also be used to assist the similarity calculation in step S304, so as to improve the reliability of the result of the similarity calculation. After the similarity is calculated in step S304, the difference between the peak values of the similarity of two sentences adjacent in position can be further calculated as the relative similarity of the sentence, or it can also be called the depth value of the sentence, so as to indicate whether the sentence is similar to the adjacent sentence. Afterwards, the paragraph division point used to divide the paragraph can be determined according to the relative similarity calculated in step S304 in step S305. For example, the sentence whose relative similarity is greater than the predetermined paragraph division threshold, that is, the sentence whose difference value of the similarity with the surrounding sentences is relatively large, can be regarded as a transition sentence, so it can be used as a paragraph division point. For example, in the embodiment of the present application, the average value and standard deviation of the relative similarity of each sentence can be further calculated, and the difference between the average value and half of the standard deviation is used as the above-mentioned predetermined paragraph division threshold. Therefore, in step S306, the original text can be divided into segmented texts based on the paragraph division points determined in this way.
[0061] In addition, after the paragraph division is performed based on the determined paragraph division points in step S306, it is likely that the paragraphs divided only based on relative similarity may contain too many paragraphs. For example, a certain paragraph contains ten or more sentences. Although the calculated similarities of these sentences are close, due to the large number of sentences, they are likely to involve two or more topics. Therefore, in the embodiment of the present application, based on the paragraphs divided in step S306, the number of sentences in each paragraph is further counted, and if the number of sentences contained in the paragraph exceeds the predetermined sentence threshold, the paragraph can be divided twice by introducing some keywords, such as "next", "then", etc. That is, the paragraph division is assisted by words that usually have the function of topic conversion, so as to ensure that the divided paragraphs have a smaller granularity.
[0062] S307, calculating the topic strength value of each sentence in the segmented text in a first predetermined number of categories.
[0063] S308, calculating the difference between the topic strength values of each sentence in each category.
[0064] S309: Use sentences whose difference is less than a predetermined difference threshold as transition sentences between the corresponding category topic and the adjacent topic.
[0065] S310, performing topic segmentation processing on the segmented text according to the transition sentences to generate topic-divided texts.
[0066] After obtaining the segmented text in step S306, since the original text has been divided into paragraphs of smaller granularity, topic clustering can be performed on the obtained segmented text. For example, various topic clustering algorithms such as LDA (Latent Dirichlet Allocation) can be used to perform clustering calculations.
[0067] For example, in step S307, the topic strength values of each sentence in each paragraph in the segmented text in a predetermined number of categories, for example, two categories, can be calculated. That is, the value of each paragraph in these two categories, that is, the topic strength value, is calculated. The topic strength value can indicate the strength of the correlation between each sentence and the topic, that is, the topic strength value can usually be expressed as a probability. For example, if the topic strength value of a sentence on the first topic is 1, it means that the sentence is exactly for the first topic. But generally speaking, the topic strength value of each sentence on different topics is usually close to 1 and 0, respectively. Next, in step S308, the difference between the topic strength values of each sentence on adjacent topics can be calculated according to the topic strength value calculated in step S307, and in step S308, the sentences whose difference is less than the predetermined difference threshold are used as transition sentences between the two topics. Specifically, since the topic strength value represents the relevance of the sentence to the topic, when the topic strength values of the sentence on two adjacent topics differ greatly, for example, the topic strength value on the first topic is 0.8, and the topic strength value on the second topic is 0.2, then the difference is 0.6, which can indicate that the sentence mainly involves one topic, namely, the first topic. On the contrary, if the topic strength value of a sentence on the first topic is 0.6, and the topic strength value on the second topic is 0.4, then the difference is 0.2, which means that this sentence is likely to be a sentence that connects the two topics, namely, a transition sentence, and therefore has a relatively close topic strength value on the two topics. Therefore, in step S309, the sentence whose difference in topic strength value is less than the predetermined difference threshold can be determined as a transition sentence between the two topics, and therefore in step S309, the topic division can be further performed based on the transition sentence determined in this way, thereby generating a sub-topic text.
[0068] In addition, in a scenario such as a classroom, teachers usually publish the syllabus of the class in advance or after the class. Therefore, when the original text of such a classroom record is subjected to the above-mentioned topic segmentation processing, multiple sub-topic sub-texts can be obtained after the segmented text is subjected to multiple topic segmentation processing based on the transition sentences determined in step S308 in step S309, and the relevance between each sub-topic sub-text and the obtained syllabus text is further calculated, so that the sub-topic sub-text with the highest relevance score is determined as the final sub-topic text. Specifically, the syllabus should be closely related to the original record of the class and the various parts of the syllabus should also correspond to the various parts of the sub-topic text determined in the above step 309. Therefore, in this case, the various transition sentences determined in step S308 can be used to perform different topic segmentation on step S309, so as to obtain parts divided by different topic segmentation points, and the relevance of such different divided parts to the various parts of the syllabus is calculated respectively, and the sum of the relevance of the various parts is calculated as the relevance of the topic segmentation text obtained by this division to the syllabus. Therefore, each transition sentence determined in step S308 can be searched in this way to find the transition sentence that is most suitable for topic segmentation. Therefore, in this way, the transition sentence that is most suitable for topic segmentation can be searched in an iterative or multiple trial manner with the aid of the introduction of the outline, thereby further improving the accuracy of topic segmentation.
[0069] In addition, when the number of categories used in the clustering calculation in step S307 is large, for example, 3 or more categories, the difference between the topic strength values calculated in step S308 will not be able to truly reflect the specific correlation between the sentence and these topics, and therefore it is difficult to use the difference as a criterion for judging the transition sentence in step S309. In other words, if the transition sentence determined in step S309 is based on the difference determined in this way, it is likely that it is not a true transition sentence between two topics, because the influence of other topics has been mixed in. Therefore, in the embodiment of the present application, in this case, the clustering calculation can be further performed for the transition sentences determined in step S309 to divide these transition sentences into multiple transition sentence sets, so that the sentence with the smallest difference in each transition sentence set can be determined as the topic division sentence for topic segmentation processing. That is, in the embodiment of the present application, by performing a secondary clustering calculation on the transition sentences determined in step S309, it is possible to make each transition sentence set involve only two topics, so that the sentence with the smallest difference in each transition sentence set can be determined as the topic division sentence for topic segmentation processing, and thus the topic division sentence determined in this way is used for topic segmentation processing. In addition, when there are too many transition sentences determined in step S309, the amount of calculation may increase due to too many attempts to calculate. Therefore, in an embodiment of the present application, when trying multiple transition sentences, a subset of all transition sentences can be used to reduce the number of transition sentences used for calculation, or the transition sentence with the greatest relevance to the current topic can be used as the center point of the attempt, and the difference between the match between the sentence and the current topic and the match between the previous or next topic is calculated as the search radius, so that the transition sentences whose relevance falls within this interval are used as the set of transition sentences to be tried, thereby reducing the amount of calculation for repeated attempts.
[0070] In addition, after the sub-topic text is determined in step S310, the perplexity between the sentences of each topic can be further calculated to confirm whether the number of topics contained in the sub-topic text is correct. For example, the perplexity between the sentences in the sub-topic text determined in step S310 can be iteratively calculated and temporarily stored as the first perplexity, and then the number of categories can be incremented based on the number of categories used in step S307, and the topic segmentation process is re-performed according to the number of incremented categories, and the sub-topic text is obtained again in step S310, and the perplexity of each sentence is calculated again to compare with the aforementioned first perplexity. If it is not greater than the first perplexity or even less than the first perplexity, it means that the number of categories used in step S307 is appropriate, and the number of topics calculated using such a number of categories is also accurate, and therefore, the sub-topic text calculated last time can be used as the final sub-topic text. On the contrary, if the perplexity calculated this time is greater than the first perplexity of the previous time, it means that the number of categories used in the previous step S307 is not suitable, so the above topic segmentation process can be repeatedly performed by increasing the number of categories until the perplexity is less than the perplexity between the sentences corresponding to the topic determined last time. Therefore, in this way, the problem of inaccurate calculation results caused by using a predetermined number of categories can be further eliminated, thereby further improving the accuracy of topic segmentation.
[0071] Therefore, by utilizing the text processing method of the embodiment of the present application, the original text can be segmented and then the positions of the sentences in the segmented text can be considered when topic clustering is performed on the generated segmented text, so that sentences that are close in time sequence have a high similarity to ensure the coherence of the sentences in terms of topic, and the text is topic segmented according to the topic clustering results to generate topic-based texts, thereby automatically providing users with texts that have been segmented and organized according to different topics, facilitating users' future searches and readings and improving user experience.
[0072] Embodiment 4
[0073] Figure 4 The schematic diagram of the text processing device embodiment provided in the present application can be used to perform the following steps: Figure 2 and Figure 3 The method steps shown. Figure 4 As shown, the text processing device may include: a first acquisition module 41, a first segmentation module 42 and a first topic segmentation module 43.
[0074] The first acquisition module 41 may be used to acquire an original text containing a plurality of sentences.
[0075] In an embodiment of the present application, an original text containing multiple sentences can be obtained based on an audio file of a target user's voice recording. For example, during a meeting, a device with an audio recording function such as a recorder can be used to record the speech of a participant, and then it can be converted into a text file through various speech recognition algorithms, thereby generating an original text corresponding to the speech of the participant of the meeting. In addition, in an embodiment of the present application, an automatic speech recognition technology can also be used to convert the recorded speech into corresponding text in real time in a scene such as a meeting, so that the original text corresponding to the speech of the participant of the meeting can be directly generated at the end of the meeting.
[0076] The first segmentation module 42 may be used to divide a plurality of sentences into at least two paragraphs according to the similarity between the sentences in the original text, so as to generate segmented text.
[0077] The original text containing multiple sentences obtained by the first acquisition module 41 is usually generated based on the speech record of the speaker in the scene such as a meeting or classroom. Therefore, such original text often has the form of a running account, especially containing various spoken words and various transition words or even sentences that connect the previous and the next. Therefore, the granularity of each sentence recorded in this way is also small, and the amount of information contained is correspondingly small. Therefore, the first segmentation module 42 can divide such a running account-style original text obtained by the first acquisition module 41 into paragraphs. For example, the first segmentation module 42 can first calculate the similarity between each sentence in the original text, and perform paragraph division processing based on the calculated similarity.
[0078] The first topic segmentation module 43 can be used to perform topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and perform topic segmentation processing on the segmented text according to the topic clustering result to generate sub-topic text.
[0079] The first topic segmentation module 43 can perform topic clustering on the segmented text obtained by the first segmentation module 42. For example, various topic clustering algorithms such as LDA (Latent Dirichlet Allocation) can be used to perform clustering calculations. In particular, in the embodiment of the present application, since the original text obtained by the first acquisition module 41 is usually more colloquial, it is very likely that two words spoken consecutively in time sequence will be regarded as unrelated when topic clustering is performed separately. Therefore, in the embodiment of the present application, by introducing the position information of the sentences in the text, it is possible to make such two sentences with different meanings have a higher similarity, thereby ensuring that the two sentences are identified as being for the same topic. Therefore, the first topic segmentation module 43 can consider the position of each sentence in the segmented text when performing topic clustering, so that adjacent sentences can have a higher correlation, thereby ensuring the coherence and sequentiality of the topic.
[0080] In addition, the text processing apparatus according to the embodiment of the present application may further include a second acquisition module 44, which may be used to acquire a topic description text corresponding to the original text.
[0081] In a scenario such as a classroom, teachers usually publish the syllabus for this class in advance or after class. This can usually be regarded as the topic description text corresponding to the original text. Therefore, when the original text of such a classroom record is subjected to the above-mentioned topic segmentation processing, the first topic segmentation module 43 can obtain multiple sub-topic sub-texts based on the topic segmentation processing of the segmented text for multiple times, and further calculate the relevance between each sub-topic sub-text and the acquired topic description text, so as to determine the sub-topic sub-text with the highest relevance score as the final sub-topic text. Specifically, for example, the topic description text of the syllabus should be closely related to the original record of the class, and the various parts of the syllabus should also correspond to the various parts of the sub-topic text determined by the first topic segmentation module 43. Therefore, in this case, the first topic segmentation module 43 can perform various topic segmentations on the segmented text to obtain parts divided by different topic segmentation points, and respectively calculate the relevance of such different divided parts with each part of the syllabus, and calculate the sum of the relevance of each part as the relevance of the topic segmentation text obtained by this division with the syllabus. Therefore, in this way, with the help of the outline, the most suitable way to segment the topic can be searched in an iterative or multiple-trial manner, thereby further improving the accuracy of the topic segmentation.
[0082] In addition, the text processing device according to the embodiment of the present application may further include a first calculation module 45 and a first merging module 46 .
[0083] The first calculation module 45 can be used to calculate the number of words in each sentence of the original text.
[0084] The first merging module 46 can be used to merge a sentence with a word count less than a predetermined word count threshold with the sentence adjacent to it in position until the number of valid words in the merged sentence that meet the predetermined criteria is greater than the predetermined valid word threshold.
[0085] In the embodiments of the present application, after obtaining the original text, since the original text is usually generated based on the speech records of speakers in scenarios such as meetings or classrooms, it is usually more colloquial in form, especially each sentence may be relatively short. For example, as Figure 1 shown, the original text may include the two sentences "This. Starting from today, we are in the next semester", and the first sentence actually only contains two words "This". Therefore, if paragraph division and subsequent topic segmentation are performed based on such sentences, due to too few words in the sentences, the granularity is too small, and the amount of information is also very small, making it difficult to be processed as a sentence unit. Therefore, in the embodiments of the present application, the first calculation module 45 can calculate the number of words in each sentence of the original text obtained by the first acquisition module 41, and the first merging module 46 can regard the sentences with the number of words calculated by the first calculation module 45 less than a predetermined word count threshold, for example, 5, as sentences with too small granularity. For example, as described here, Figure 1 in the original text shown, the sentence "This" contains only two words, so it obviously belongs to a sentence with too small granularity. Therefore, the first merging module 46 can merge this sentence with the sentence adjacent to it in position, that is, merge it with the subsequent sentence "Starting from today, we are in the next semester" into one sentence. Thus, the number of words contained in the merged sentence is obviously greater than 5, so it can be used as a sentence unit for subsequent paragraph division and topic division. In addition, in the embodiments of the present application, the number of valid words that meet the predetermined criteria in the merged sentence can also be further counted. For example, in the sentence "Starting from today, we are in the next semester" mentioned above, words other than modal particles such as "呢", "了", and "啊" can be regarded as valid words, and the number of such valid words can be counted to determine whether further merging with adjacent sentences is needed. Therefore, through such processing, it can be ensured that the sentence units for subsequent paragraph division and topic processing contain enough meaningful words, which can further ensure the accuracy and efficiency of the calculation. In addition, in the embodiments of the present application, whether a word is a stop word can also be used to determine valid words, that is, each sentence in the original text is screened according to a predetermined list or database of stop words to regard the words other than the removed stop words as valid words.
[0086] In addition, according to the embodiment of the present application, the first segmentation module 42 may further include a similarity calculation unit 421 , a paragraph division unit 422 and a segment text generation unit 423 .
[0087] The similarity calculation unit 421 may be configured to calculate, based on the similarity of each sentence, a difference between peak values of similarity between each sentence and at least two sentences adjacent to the sentence in the original text as the relative similarity of the sentence.
[0088] The paragraph division unit 422 may be configured to determine paragraph division points for dividing the original text into paragraphs according to the relative similarity.
[0089] The segmented text generating unit 423 may be configured to generate segmented text based on paragraph division points.
[0090] The similarity calculation unit 421 can calculate the similarity between adjacent sentences for the original text containing multiple sentences acquired by the first acquisition module 41 or the text merged by the first merging module 46. For example, in the present application, the cosine value can be used to calculate the similarity between adjacent sentences. In addition, in the embodiment of the present application, the similarity calculation unit 421 can also use some pre-trained word vectors to assist in similarity calculation, so as to improve the reliability of the result of similarity calculation. The similarity calculation unit 421 can further calculate the difference between the peak values of the similarity of two sentences adjacent in position as the relative similarity of the sentence, or it can also be called the depth value of the sentence, so as to indicate whether the sentence is similar to the adjacent sentence. Afterwards, the paragraph division unit 422 can determine the paragraph division point used to divide the paragraph based on the relative similarity calculated by the similarity calculation unit 421. For example, a sentence whose relative similarity is greater than a predetermined paragraph division threshold, that is, a sentence whose similarity difference with the surrounding sentences is relatively large, can be regarded as a transition sentence, and therefore it can be used as a paragraph division point. For example, in the embodiment of the present application, the average value and standard deviation of the relative similarity of each sentence can be further calculated, and the difference between the average value and half of the standard deviation is used as the above-mentioned predetermined paragraph division threshold. Therefore, the segmented text generation unit 423 can divide the original text into segmented texts based on the paragraph division points determined in this way.
[0091] In addition, after the paragraph division unit 422 divides the paragraphs, it is likely that the paragraphs divided only based on relative similarity may contain too many paragraphs. For example, a certain paragraph contains ten or more sentences. Although the calculated similarities of these sentences are close, due to the large number of sentences, they are likely to involve two or more topics. Therefore, in the embodiment of the present application, the text processing device may further include a second calculation module 47, which can further count the number of sentences in each paragraph based on the paragraphs divided by the paragraph division unit 422, and the first segmentation module 42 can introduce some keywords, such as "next", "then", etc., to divide the paragraph twice when the number of sentences contained in the paragraph exceeds the predetermined sentence threshold. That is, the words that usually have the function of topic conversion are used to assist in paragraph division, so as to ensure that the divided paragraphs have a smaller granularity.
[0092] In addition, in the embodiment of the present application, the first topic segmentation module 43 may further include a topic strength value calculation unit 431 , a difference calculation unit 432 , a transition sentence determination unit 433 and a topic segmentation unit 434 .
[0093] The topic strength value calculation unit 431 may be configured to calculate the topic strength value of each sentence in the segmented text in a first predetermined number of categories.
[0094] The difference calculation unit 432 may be configured to calculate the difference between the topic strength values of each sentence in each category.
[0095] The transition sentence determining unit 433 may be configured to use a sentence whose difference is less than a predetermined difference threshold as a transition sentence between a corresponding category topic and an adjacent topic.
[0096] The topic segmentation unit 434 may be configured to perform topic segmentation processing on the segmented text according to the transition sentences to generate sub-topic texts.
[0097] After the first segmentation module 42 obtains the segmented text, since the original text has been divided into paragraphs of smaller granularity, topic clustering can be performed on the obtained segmented text. For example, various topic clustering algorithms such as LDA (Latent Dirichlet Allocation) can be used to perform clustering calculations.
[0098] For example, the topic strength value calculation unit 431 can calculate the topic strength values of each sentence in each paragraph in the segmented text in a predetermined number of categories, for example, two categories. That is, the value of each paragraph in these two categories, that is, the topic strength value, is calculated. The topic strength value can indicate the strength of the correlation between each sentence and the topic, that is, the topic strength value can usually be expressed as a probability. For example, if the topic strength value of a sentence on the first topic is 1, it means that the sentence is exactly for the first topic. But generally speaking, the topic strength value of each sentence on different topics is usually close to 1 and 0, respectively. Next, the difference calculation unit 432 can calculate the difference between the topic strength values of each sentence on adjacent topics according to the topic strength value calculated by the topic strength value calculation unit 431, and the transition sentence determination unit 433 can use the sentence whose difference is less than the predetermined difference threshold as the transition sentence between the two topics. Specifically, since the topic strength value represents the relevance of the sentence to the topic, when the topic strength values of the sentence on two adjacent topics are very different, for example, the topic strength value on the first topic is 0.8, and the topic strength value on the second topic is 0.2, then the difference is 0.6, which can indicate that the sentence mainly involves one topic, namely the first topic. On the contrary, if the topic strength value of a sentence on the first topic is 0.6, and the topic strength value on the second topic is 0.4, then the difference is 0.2, which means that this sentence is likely to be a sentence that connects the two topics, namely a transition sentence, and therefore has a relatively close topic strength value on the two topics. Therefore, the transition sentence determination unit 433 can determine such a sentence whose difference in topic strength value is less than a predetermined difference threshold as a transition sentence between the two topics, and therefore the topic segmentation unit 434 can further perform topic division based on the transition sentences determined in this way, thereby generating sub-topic texts.
[0099] In addition, when the number of categories used in clustering calculation by the first topic segmentation module 43 is large, for example, 3 or more categories, the difference between the topic strength values calculated by the topic strength value calculation unit 431 will not be able to truly reflect the specific correlation between the sentence and these topics, and therefore it is difficult for the transition sentence determination unit 433 to use the difference as a criterion for judging the transition sentence. In other words, if the topic segmentation unit 434 determines the transition sentence based on the difference determined in this way, it is very likely that it is not a true transition sentence between two topics because the influence of other topics has been mixed in. Therefore, in this embodiment of the present application, in this case, a clustering calculation can be further performed on the transition sentences determined by the transition sentence determination unit 433 to divide these transition sentences into multiple transition sentence sets, so that the sentence with the smallest difference in each transition sentence set can be determined as the topic division sentence for topic segmentation processing. That is, in the embodiment of the present application, by performing a secondary clustering calculation on the transition sentences determined by the transition sentence determination unit 433, each transition sentence set can be made to involve only two topics, so that the sentence with the smallest difference in each transition sentence set can be determined as the topic division sentence for topic segmentation processing, and thus the topic division sentence determined in this way is used to perform topic segmentation processing. In addition, when there are too many transition sentences determined by the transition sentence determination unit 433, the amount of calculation may increase due to too many attempts to calculate. Therefore, in the embodiment of the present application, when trying multiple transition sentences, a subset of all transition sentences can also be used to reduce the number of transition sentences used for calculation, or the transition sentence with the greatest relevance to the current topic can be used as the center point of the attempt, and the difference between the matching degree of the sentence with the current topic and the matching degree of the previous or next topic is calculated as the search radius, so that the transition sentences with the relevance falling within this interval are used as the transition sentence set to be tried, thereby reducing the amount of calculation for repeated attempts.
[0100] In addition, the text processing device of the embodiment of the present application may further include a third calculation module 48, which can further calculate the perplexity between the sentences of each topic after the first topic segmentation module 43 determines the sub-topic text to confirm whether the number of topics contained in the sub-topic text is correct. For example, the perplexity between the sentences in the sub-topic text determined by the first topic segmentation module 43 can be determined in an iterative manner, and temporarily stored as the first perplexity, and then the number of categories can be incremented based on the number of categories used by the first segmentation module 42, and the topic segmentation process can be re-performed according to the number of incremented categories, and the sub-topic text is obtained again through the first topic segmentation module 43, and the perplexity of each sentence is calculated again by the third calculation module 48 to compare with the aforementioned first perplexity. If it is not greater than the first perplexity or even less than the first perplexity, it means that the number of categories used by the first segmentation module is appropriate, and the number of topics calculated using such a number of categories is also accurate, and therefore, the sub-topic text calculated last time can be used as the final sub-topic text. On the contrary, if the perplexity calculated this time is greater than the first perplexity of the previous time, it means that the number of categories used by the first segmentation module 42 of the previous time is not appropriate, so the above topic segmentation process can be repeatedly performed by increasing the number of categories until the perplexity is less than the perplexity between the sentences corresponding to the topic determined last time. Therefore, in this way, the problem of inaccurate calculation results caused by using a predetermined number of categories can be further eliminated, thereby further improving the accuracy of topic segmentation.
[0101] Therefore, by utilizing the text processing device of the embodiment of the present application, the original text can be segmented and then the positions of the sentences in the segmented text can be considered when topic clustering is performed on the generated segmented text, so that sentences that are close in time sequence have a high similarity to ensure the coherence of the sentences in terms of topic, and the text is topic segmented according to the topic clustering results to generate topic-based texts, thereby automatically providing users with texts that have been segmented and organized according to different topics, facilitating users' future searches and readings and improving user experience.
[0102] Embodiment 5
[0103] The internal functions and structure of the text processing device are described above. The device can be implemented as an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device embodiment provided by this application. Figure 5 As shown, the electronic device includes a memory 51 and a processor 52 .
[0104] The memory 51 is used to store programs. In addition to the above programs, the memory 51 can also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0105] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0106] The processor 52 is not limited to a central processing unit (CPU), but may also be a processing chip such as a graphics processing unit (GPU), a field programmable gate array (FPGA), an embedded neural network processor (NPU) or an artificial intelligence (AI) chip. The processor 52 is coupled to the memory 51 to execute the program stored in the memory 51. When the program is running, the text processing method of the above-mentioned embodiments 2 and 3 is executed.
[0107] Further, if Figure 5 As shown, the electronic device may also include: a communication component 53, a power component 54, an audio component 55, a display 56 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.
[0108] The communication component 53 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 53 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 53 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0109] The power supply component 54 provides power to various components of the electronic device. The power supply component 54 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.
[0110] The audio component 55 is configured to output and / or input audio signals. For example, the audio component 55 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 51 or sent via the communication component 53. In some embodiments, the audio component 55 also includes a speaker for outputting audio signals.
[0111] The display 56 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0112] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A text processing method, comprising: Get the original text containing multiple sentences; Dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original text to generate segmented text; The generated segmented text is topic clustered in combination with the position information of each sentence in the segmented text, and the segmented text is topic segmented according to the topic clustering result to generate sub-topic text; wherein, the generated segmented text is topic clustered in combination with the position information of each sentence in the segmented text, and the segmented text is topic segmented according to the topic clustering result to generate sub-topic text, including: calculating the topic strength value of each sentence in the segmented text in a first predetermined number of categories; calculating the difference between the topic strength values of each sentence in each category; using sentences with differences less than a predetermined difference threshold as transition sentences between the corresponding category topic and the adjacent topic; and performing topic segmentation on the segmented text according to the transition sentences to generate sub-topic text.
2. The text processing method according to claim 1, wherein: The method further comprises: Obtaining a topic description text corresponding to the original text, and The step of performing topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and performing topic segmentation processing on the segmented text according to the topic clustering result to generate topic-divided text includes: Text matching processing is performed on the topic description text and the topic clustering result, and sub-topic texts are generated according to the text matching processing result.
3. The text processing method according to claim 1, wherein: Before dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original text to generate segmented text, the text processing method further comprises: Calculate the number of words in each sentence in the original text; A sentence whose word count is less than a predetermined word count threshold is merged with a sentence adjacent to the sentence in position, until the number of valid words meeting a predetermined standard in the merged sentence is greater than a predetermined valid word threshold.
4. The text processing method according to claim 3, wherein: The valid words are determined by removing stop words in the sentence.
5. The text processing method according to claim 1, wherein: The step of dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original text to generate segmented texts includes: Calculate the difference between the peak values of the similarity between each sentence and at least two sentences adjacent to the sentence in the original text according to the similarity of each sentence as the relative similarity of the sentence; Determining paragraph division points for dividing the original text into paragraphs according to the relative similarity; Segmented text is generated based on the paragraph division points.
6. The text processing method according to claim 5, wherein: The step of determining the paragraph division points for dividing the original text into paragraphs according to the relative similarity comprises: Calculate the mean and standard deviation of the relative similarity of each sentence; Calculate the difference between the mean value and half of the standard deviation as a paragraph division threshold; The sentences whose relative similarity is greater than the paragraph division threshold are used as the paragraph division points.
7. The text processing method according to claim 1, wherein: After dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original text to generate segmented text, the text processing method further comprises: Counting the number of sentences contained in each paragraph of the segmented text; For a paragraph whose number of sentences is greater than a predetermined sentence threshold, the paragraph is further divided into paragraphs according to predetermined keywords.
8. The text processing method according to claim 1, wherein: The topic segmentation processing of the segmented text according to the transition sentence to generate sub-topic texts includes: Using each transition sentence as a topic division sentence, the segmented text is subjected to topic segmentation processing to generate a plurality of topic sub-texts; Calculate the relevance between each sub-topic sub-text and the predetermined outline text; The sub-topic sub-text with the highest relevance is used as the sub-topic text.
9. The text processing method according to claim 1, wherein: The method further comprises: Calculate the perplexity between sentences of each topic as the first perplexity; gradually increasing the number of categories used for the topic clustering, and re-clustering the topics according to the increased number of categories; The topics of the original text and the perplexity between the corresponding sentences are determined again according to the results of the re-conducted topic clustering, until the perplexity is less than the perplexity between the sentences corresponding to the topics determined last time.
10. The text processing method according to claim 1, wherein: The topic segmentation processing of the segmented text according to the transition sentence to generate sub-topic texts includes: Performing a second clustering calculation on transition sentences between two topics to generate multiple transition sentence sets; Determine the sentence with the smallest difference in each transition sentence set as the topic division sentence to be processed for topic segmentation; The segmented text is processed by topic segmentation according to the topic division sentences to generate topic-divided texts.
11. A text processing device, comprising: A first acquisition module is used to acquire an original text containing a plurality of sentences; A first segmentation module, configured to divide the plurality of sentences into at least two paragraphs according to the similarity between the sentences in the original text, so as to generate segmented text; The first topic segmentation module is used to perform topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and perform topic segmentation processing on the segmented text according to the topic clustering result to generate sub-topic text; wherein, the topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and perform topic segmentation processing on the segmented text according to the topic clustering result to generate sub-topic text includes: calculating the topic strength value of each sentence in the segmented text in a first predetermined number of categories; calculating the difference between the topic strength values of each sentence in each category; using sentences with differences less than a predetermined difference threshold as transition sentences between the corresponding category topic and the adjacent topic; and performing topic segmentation processing on the segmented text according to the transition sentences to generate sub-topic text.
12. A text processing method, comprising: Get the original classroom transcript containing multiple sentences; Dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the original classroom records to generate a segmented text of the classroom records; The generated classroom record segmented text is topic clustered in combination with the position information of each sentence in the classroom record segmented text, and the classroom record segmented text is topic segmented according to the topic clustering result to generate sub-topic text; wherein, the generated classroom record segmented text is topic clustered in combination with the position information of each sentence in the classroom record segmented text, and the classroom record segmented text is topic segmented according to the topic clustering result to generate sub-topic text, including: calculating the topic strength value of each sentence in the classroom record segmented text in a first predetermined number of categories; calculating the difference between the topic strength values of each sentence in each category; using sentences with differences less than a predetermined difference threshold as transition sentences between the corresponding category topic and the adjacent topic; and performing topic segmentation on the classroom record segmented text according to the transition sentences to generate sub-topic text.
13. The text processing method according to claim 12, wherein: The method further comprises: Obtaining a course outline corresponding to the original classroom record, wherein the course outline includes a plurality of course content points, and The step of performing topic clustering on the generated classroom record segmented text in combination with the position information of each sentence in the classroom record segmented text, and performing topic segmentation processing on the classroom record segmented text according to the topic clustering result to generate topic-divided text includes: Text matching processing is performed on the topic description text and the topic clustering result, and sub-topic texts are generated according to the text matching processing results, wherein each sub-topic text corresponds to a course content key point among the multiple course content key points.
14. The text processing method according to claim 13, wherein: After obtaining the course outline corresponding to the original classroom record, the method further includes: The course content main points of the course outline are aligned with the topics of the topic clusters according to the keywords of each category of the topic clusters.
15. The text processing method according to claim 12, wherein: The obtaining of the original classroom record containing multiple sentences includes: The teaching voice is acquired from the teaching audio through the voice recognition technology, and the acquired teaching voice is converted into a plurality of sentences.
16. A text processing method, comprising: Get meeting transcripts containing multiple sentences; Dividing the plurality of sentences into at least two paragraphs according to the similarities between the sentences in the meeting minutes to generate segmented text; The generated segmented text is clustered based on the position information of each sentence in the segmented text, and the segmented text is segmented according to the topic clustering result to generate sub-topic text; wherein, the generated segmented text is clustered based on the position information of each sentence in the segmented text, and the segmented text is segmented according to the topic clustering result to generate sub-topic text, including: calculating the topic strength value of each sentence in the segmented text in a first predetermined number of categories; calculating the difference between the topic strength values of each sentence in each category; using sentences with differences less than a predetermined difference threshold as transition sentences between the corresponding category topic and the adjacent topic; and performing topic segmentation on the segmented text according to the transition sentences to generate sub-topic text.
17. The text processing method according to claim 16, wherein: The method further comprises: Generate the topic points corresponding to each topic in the sub-topic text according to the sentences contained in the topic.
18. The text processing method according to claim 16, wherein: The method further comprises: obtaining an agenda corresponding to the meeting minutes, wherein the agenda includes a plurality of topics of the meeting corresponding to the meeting minutes, and The step of performing topic clustering on the generated segmented text in combination with the position information of each sentence in the segmented text, and performing topic segmentation processing on the segmented text according to the topic clustering result to generate topic-divided texts includes: Text matching is performed on the agenda and the topic clustering results, and sub-topic texts are generated according to the text matching results.
19. The text processing method according to claim 16, wherein: The obtaining of the meeting record containing multiple sentences includes: The voice of at least one conference participant is acquired from the conference audio by using a voice recognition technology, and the acquired voice of the conference participant is converted into a plurality of sentences.
20. The text processing method according to claim 19, wherein: The obtaining of the meeting record containing a plurality of sentences further comprises: When multiple conference participants are identified, identity tags of corresponding conference participants are added to sentences corresponding to the voices of the identified conference participants respectively.
21. An electronic device, wherein: include: Memory, used to store programs; A processor is used to run the program stored in the memory, and the program executes the text processing method as claimed in any one of claims 1 to 10 and 12 to 20 when it is run.
22. A computer-readable storage medium having stored thereon a computer program executable by a processor, wherein: When the program is executed by a processor, the text processing method according to any one of claims 1 to 10 and 12 to 20 is implemented.
