Semantic fragmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of natural language processing, and discloses a semantic fragmentation method and device, electronic equipment and a storage medium. According to the method, sentences in a to-be-processed text are clustered by applying a hierarchical clustering algorithm and a position reward mechanism, and preliminary semantic fragments are formed; searching semantic breakpoints of the to-be-processed text by comparing the distances between the embedded vectors corresponding to the sentences in the to-be-processed text; and obtaining a final semantic fragment of the to-be-processed text in combination with the preliminary semantic fragment and the semantic breakpoint. The problem of semantic information loss caused by simple rule fragmentation is effectively avoided, and the accuracy and robustness of fragmentation are greatly improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of natural language processing, and in particular to a semantic segmentation method, device, electronic device and storage medium. Background Art

[0002] In Natural Language Processing (NLP), text segmentation is a basic step in tasks such as text analysis, information retrieval, and machine translation. Related text segmentation technologies can be mainly divided into rule-based methods and machine learning-based methods, but both currently have their own shortcomings.

[0003] The inventors have discovered that there are at least the following problems in the related art: the defects of the current text segmentation technology mean that text segmentation still faces challenges under complex semantic structures, and there is an urgent need for a more sophisticated and robust segmentation method to fill the relevant technical gaps in this field. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a semantic sharding method, device, electronic device and storage medium, so as to avoid the loss of semantic information while greatly improving the accuracy and robustness of sharding.

[0005] To solve the above technical problems, an embodiment of the present invention provides a semantic segmentation method, comprising: clustering sentences in a text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic segmentation; finding semantic breakpoints of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; and combining the preliminary semantic segmentation and the semantic breakpoints to obtain the final semantic segmentation of the text to be processed.

[0006] An embodiment of the present invention also provides a semantic segmentation device, comprising: an application clustering module, used to cluster sentences in a text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic segmentations; a semantic breakpoint module, used to find the semantic breakpoints of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; and a semantic processing module, used to combine the preliminary semantic segmentations and the semantic breakpoints to obtain the final semantic segmentations of the text to be processed.

[0007] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned semantic segmentation method.

[0008] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned semantic sharding method when executed by a processor.

[0009] In the embodiment of the present invention, the sentences in the text to be processed are clustered by applying a hierarchical clustering algorithm and a position reward mechanism to form a preliminary semantic segmentation; the semantic breakpoints of the text to be processed are found by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; and the final semantic segmentation of the text to be processed is obtained by combining the preliminary semantic segmentation and the semantic breakpoints. The problem of semantic information loss caused by simple rule segmentation is effectively avoided, and the accuracy and robustness of the segmentation are greatly improved.

[0010] In addition, the method of finding the semantic breakpoint of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed includes: forming multiple windows by combining the sentences in the text to be processed, each of which contains one or more sentences; respectively calculating the embedding vectors corresponding to each of the windows; comparing the distances between the embedding vectors corresponding to each adjacent window in turn by comparing them two by two, and obtaining a dynamic semantic difference set; finding the semantic breakpoint of the text to be processed by using the dynamic semantic difference set. To further reduce the noise caused by the embedding of a single sentence, a window mechanism is used, for example, every three sentences are used as a window, the window embedding is calculated, and the embedding distance is compared. The stability and accuracy of the segmentation process are effectively improved.

[0011] In addition, the method of finding the semantic breakpoints of the text to be processed through the dynamic semantic difference set includes: based on a preset threshold, according to the change in the distance between the embedding vectors corresponding to each of the adjacent windows in the dynamic semantic difference set, calculating the peak cutting points under the threshold; when the calculated peak cutting points are not in the preset sharding range, based on the peak decreasing strategy, by lowering the threshold and calculating the peak cutting points under the threshold again according to the dynamic semantic difference set until the peak cutting points are in the preset sharding range; when the calculated peak cutting points are in the preset sharding range, marking the position where the distance between the embedding vectors corresponding to each of the adjacent windows exceeds the threshold as a semantic breakpoint, and finding the position with a larger embedding distance as a semantic breakpoint. By setting a threshold, the position with a larger distance is marked as the beginning of a new semantic paragraph. The window mechanism and the walking method are used to significantly reduce the influence of noise in the embedding vector calculation, ensuring that the sharding results are more stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0013] Figure 1 is a flow chart of a semantic sharding method provided according to an embodiment of the present invention;

[0014] Figure 2 A hierarchical clustering tree diagram according to an embodiment of the present invention

[0015] Figure 3 is a structural schematic diagram of a semantic sharding device according to another embodiment of the present invention;

[0016] Figure 4 It is a schematic structural diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0018] One embodiment of the present invention relates to a semantic segmentation method, which can be applied to computer devices capable of performing natural language classification processing, such as mobile phones, computers, etc. In this embodiment, the sentences in the text to be processed are clustered by applying a hierarchical clustering algorithm and a position reward mechanism to form a preliminary semantic segmentation; the semantic breakpoints of the text to be processed are found by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; the final semantic segmentation of the text to be processed is obtained by combining the preliminary semantic segmentation and the semantic breakpoints. The problem of semantic information loss caused by simple rule segmentation is effectively avoided, and the accuracy and robustness of the segmentation are greatly improved. The implementation details of the semantic segmentation method of this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding, and is not necessary for the implementation of this solution.

[0019] like Figure 1 As shown, in step 101, the computer device clusters the sentences in the text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic fragments.

[0020] Among them, the sentences in the text to be processed can be obtained after pre-processing and segmenting the text to be processed based on predefined rules. For example, text pre-processing is to identify the period in the text to be processed and segment the text content of the text to be processed at the period position to obtain the sentences in each text to be processed. Specifically, it can be: using the end-of-sentence symbols such as the period, question mark, exclamation mark as separators to divide the entire text content into relatively independent sentences. The purpose of this optional step is to break down continuous text paragraphs into smaller units for subsequent processing and analysis.

[0021] In an example, the text to be processed that needs to be clustered may contain advertisements, emoticons, special characters, hyperlinks and other contents. Advertisements and other contents can be eliminated through regular expressions in the early stage to avoid greatly affecting the effect of text clustering. Some texts have too few characters and cannot present complete semantic information, which will affect the clustering accuracy. Such texts also need to be eliminated by setting rules. At this time, the regular expressions in the preprocessing of the text to be processed based on predefined rules are used to eliminate the original text data containing advertisements, emoticons, special characters and hyperlinks.

[0022] In one example, in step 101, the sentences in the text to be processed are clustered by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic fragments, which can be: converting the sentences in the text to be processed into corresponding high-dimensional vector representations to capture the semantic information in the sentence units; then adding similarity scores to the high-dimensional vector representations corresponding to adjacent sentences in the text to be processed by the position reward mechanism in the application of the hierarchical clustering algorithm to form the preliminary semantic fragments; wherein the similarity scores are used to increase the probability of adjacent sentences in the text to be processed being clustered into the same cluster during the process of clustering the high-dimensional vector representations by applying the hierarchical clustering algorithm, thereby avoiding increasing the computational complexity of sentences with a smaller number of texts and negatively affecting the clustering results.

[0023] In one example, the number of texts to be processed after preprocessing (such as the number of words, characters or words, etc., which can be set according to user needs) is counted to determine the data volume of the target clustered text, and to determine whether it is necessary to use a hierarchical clustering algorithm to split the text to be processed. For example, if the number of target clustered texts is relatively small, the text to be processed can be directly segmented based on predefined rules to reduce resource consumption. The text quantity threshold for determining whether the clustered text needs to be split can be set manually. For example, the default text quantity threshold is 5000. When the number of texts to be clustered exceeds the set threshold, the hierarchical clustering algorithm in the machine learning method is used to cluster the high-dimensional vectors of these sentences, and the semantic segmentation of the text is preliminarily completed to ensure the processing accuracy.

[0024] Hierarchical clustering refers to an algorithm that divides a data set into layers of clusters, and the clusters generated in the next layer are based on the results of the previous layer. The more commonly used method is agglomerative hierarchical clustering, referred to as AGNES algorithm. Hierarchical clustering is a recursive clustering algorithm that can show the hierarchical relationship between sentences by constructing a tree structure (i.e., a dendrogram). In an example, when the number of sentences in the text to be processed exceeds the text quantity threshold, the text to be processed needs to be split, and the text to be processed is split into multiple groups for clustering respectively. The number of sentences contained in each group can be set manually, for example, the number of sentences in each group is set between 1000-5000, for example, 3000 is selected. In this way, the number of sentences in each group will not increase the amount of calculation due to being too high, and the number of texts in the group will not be too low, resulting in too few cluster samples in a single group, which may cause the problem of failing to form effective clusters and affecting the accuracy of clustering results. The number of final split clusters will be automatically determined based on the total amount of target clustered text and the amount of text contained in a single cluster. For example, if the total amount of text to be clustered is 300,000 and the amount of text contained in each cluster is 3,000, the algorithm will automatically split it into 100 clusters.

[0025] Among them, the sentences in the text to be processed are converted into corresponding high-dimensional vector representations. Since sentence embedding technology is used, and sentence embedding technology combines semantic information to process the sentences in the text to be processed, the obtained high-dimensional vector representation can capture the semantic information in the sentence unit, making the vector distance of similar sentences closer and the vector distance of sentences with different meanings farther. The high-dimensional vectors of these sentences are clustered using a hierarchical clustering algorithm to preliminarily complete the semantic segmentation of the text.

[0026] The process of clustering the text to be processed by a hierarchical clustering algorithm, that is, calculating the similarity between different samples, clustering based on the text similarity, and finally obtaining the clustering result. In an example, if the text to be processed is divided into multiple groups and clustered separately, since the text clustering in different groups is performed independently, the text clustering process of different groups can be performed synchronously, supporting the improvement of computing efficiency through multi-process computing or distributed computing. Finally, all text clusters in different groups are merged into the same large group. Similarly, based on the similarity between text clusters, the text clusters in the large group are further merged to obtain the preliminary semantic segmentation result of the text to be processed.

[0027] In the merging algorithm of the hierarchical clustering algorithm, the similarity between two types of data points is calculated, and the two most similar data points among all data points are combined, and this process is repeated. Simply put, the merging algorithm of the hierarchical clustering is to determine the similarity between them by calculating the distance between each category of data points and all data points. The smaller the distance, the higher the similarity. The two data points or categories with the closest distance are combined to generate a clustering tree. Figure 2 As shown, in one example, the Euclidean distance method can be used to calculate the distance (similarity) between data points of different categories, that is, the Euclidean distance method is used to calculate the distance between A, F, B, C, D, E, and G, and the data with the highest similarity are merged into a cluster. The method for calculating the distance between two combined data points can be the shortest distance method (Single Linkage), the absolute distance method (Complete Linkage), and the average distance method (Average Linkage), that is, the similarity between (A, F), (B, C), (D, E), and G is calculated, wherein G is used as a separate data point to calculate the distance with a combined data point such as (A, F), and the Euclidean distance method can also be used to calculate. When calculating the distance between a combined data point such as (A, F) and a separate data point such as G, the average distance from A to G and from F to G needs to be calculated respectively. After calculating the similarity between the combined data points, the data with the highest similarity can also be merged into a cluster, and multiple combined data points can also be further merged to generate a clustering tree.

[0028] After the text is clustered, multiple text clusters will be formed. The number of clustered text clusters is related to the text similarity setting during clustering. In the step of adding similarity scores to the high-dimensional vector representations corresponding to adjacent sentences in the text to be processed through the position reward mechanism, specifically, it can be: by introducing the position reward mechanism, additional similarity scores are added to adjacent sentences on the basis of calculating the similarity. This behavior will increase the similarity between adjacent sentences, making it easier for them to form the same cluster during the hierarchical clustering process, and realize the preliminary semantic segmentation of the text. Among them, the position reward mechanism can ensure that adjacent sentences are more likely to be clustered together in the hierarchical clustering method. This is because the position reward mechanism adds additional similarity scores to adjacent sentences, and the similarity scores will affect the similarity in the hierarchical clustering process. In other words, the similarity scores here will be overwritten after being merged with the similarity calculated in the hierarchical clustering process, so that the sentences corresponding to these corrected similarities will be easier to form the same cluster during the clustering process.

[0029] Among them, the additional similarity score can be a preset value added only to adjacent sentences. Or the additional similarity score can change with the length of the corresponding sentence. Since in the process of semantic segmentation, short sentences in the text may significantly change the semantics of the fragment and affect the segmentation effect, in one example, on the basis of proximity, the additional similarity score is related to the length of the sentence. The shorter the sentence, the greater the additional similarity score, making it more likely that the short sentence can be clustered into the same cluster with the adjacent sentences, reducing the negative impact of the short sentence on the segmentation results.

[0030] In one example, in the clustering results, there will be some text clusters with relatively small numbers of texts, which can even be 1 (that is, the text cluster contains only one text, and the similarity between the text and other cluster texts is less than the set threshold). In the subsequent process of merging text clusters of different groups, the text clusters with small numbers of such texts will increase the computational complexity. Therefore, you can choose to eliminate the text clusters with less text than the set threshold in this step. Although a higher set threshold will improve the computational efficiency, it is more likely to eliminate valuable clustering data and affect the accuracy of the final clustering results. Therefore, the threshold should be set prudently and not too high. The default setting value is 2, which means that text clusters with less than 2 texts in the text cluster are eliminated.

[0031] In one example, the method of converting sentences in the text to be processed into corresponding high-dimensional vector representations can be: using a pre-trained text embedding model to convert each sentence in the text to be processed into a set of high-dimensional vector representations. The pre-trained text embedding model can be a related model for generating word vectors (abbreviated as WordToVec) or a word vector model (abbreviated as GloVe, full name Global Vectors for Word Representation) model. The pre-trained text embedding model is trained by adopting a preset training strategy to capture the semantic information in the sentence unit and obtain a high-dimensional vector representation.

[0032] Compared with the conventional machine learning method, that is, the method of converting sentences into high-dimensional vectors by using embedding technology (for example, using a text representation vector model, also known as "Embedding Models"), and performing segmentation by calculating the similarity between vectors, the quality of the vectors in the segmentation method based on embedding in this application does not depend on the training data and model, and is therefore less susceptible to noise; on the other hand, in the case of a mixture of long and short sentences in conventional machine learning methods, short sentences may significantly change the semantics of the fragments and affect the segmentation effect. For example, directly using the embedding of each sentence to calculate the similarity may cause errors due to the influence of short sentences. The present application performs hierarchical clustering through sentence embedding and introduces position rewards, so that adjacent sentences are more likely to be clustered into the same cluster. This allows some short sentences to be clustered into long sentences, effectively avoiding the problem of semantic information loss caused by segmentation using simple preset rule-based methods, and reducing the negative impact of short sentences on segmentation results in machine learning methods.

[0033] In step 102, the computer device finds the semantic breakpoints of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed. For example, the sentences in the text to be processed can be combined to form multiple windows, each of which contains one or more sentences, thereby avoiding the noise caused by the embedding of a large number of single sentences to a certain extent; then, the dynamic semantic difference set of the text to be processed is obtained by calculating and comparing the embedding vectors corresponding to each window; finally, the semantic breakpoints of the text to be processed are found through the dynamic semantic difference set.

[0034] It can be seen that a window containing N sentences needs to be defined in the window mechanism. The combination of windows can be to start from the first sentence of the text and capture N sentences as a window, where N is an arbitrary positive integer. In one example, the value of the number of sentences N contained in different windows is related to the length of the sentences contained, rather than the value of the number of sentences N contained in each window being the same. Of course, the same value can also be taken for the number of sentences N contained in each window. In some cases, the number of texts contained in each window can be limited, or, in some cases, windows are allocated preferentially to texts that do not meet the preset number range to avoid relatively too few or too many texts in a certain window, so that sentences with fewer texts will not appear alone in a window as much as possible, so as to avoid texts with fewer texts increasing the computational complexity and affecting the final result.

[0035] In one example, by calculating and comparing the embedding vectors corresponding to each window, the dynamic semantic difference set of the text to be processed can be obtained by: calculating the embedding vectors corresponding to each window respectively; then comparing the distances between the embedding vectors corresponding to each adjacent window in turn by comparing them two by two, and obtaining the dynamic semantic difference set. The method of integrating some sentences and comparing them two by two using the window mechanism can significantly reduce the noise impact in the embedding vector calculation, making the final result more accurate.

[0036] Specifically, by comparing the distances between the embedding vectors corresponding to each adjacent window in turn, for example, starting from the first window, the distances between the embedding vectors in adjacent windows are compared sentence by sentence. The specific operation is to compare the embedding vector of the first window with the embedding vector of the second window, and then compare the embedding vector of the second window with the embedding vector of the third window, and so on. Through this step-by-step comparison method, the distance changes of the embedding vectors between different windows can be identified. These distance change degrees can be considered as a collection of dynamic semantic differences, reflecting the degree of semantic change between different windows in the text.

[0037] It is worth noting that if there is only one sentence in all windows, embedding calculations are performed on the sentences in the window to obtain their high-dimensional vector representations, which can be regarded as converting each sentence in the text to be processed into its corresponding high-dimensional vector representation in turn. This indicates that the characteristics of each sentence in the text are obvious, and the noise caused by the embedding of a single sentence is extremely small. At this time, there is no need to divide the window. The walking method can be used directly starting from the first sentence of the text, and the distance between the sentence embeddings is calculated sentence by sentence. The position where the distance exceeds the preset threshold is marked as a semantic breakpoint. Since there is no need to divide the window for each sentence, the system can save the computing resources required for window allocation.

[0038] In one example, the embedding vectors corresponding to each window are calculated separately, which can be: embedding calculations are performed on the sentences contained in the window in turn to obtain high-dimensional vector representations of the sentences contained in the window; high-dimensional vector representations of the sentences contained in the same window are combined and calculated by a matrix algorithm to obtain the embedding vector corresponding to the window. That is, embedding calculations are performed on the sentences in the window to obtain their high-dimensional vector representations, and these vectors are combined into an overall high-dimensional vector through a matrix algorithm. The overall high-dimensional vector contains the semantics of the N sentences in this window. The purpose of this step is to reduce the noise caused by the embedding of a single sentence and to improve stability by combining the semantic information of multiple sentences.

[0039] In one example, the semantic breakpoints of the text to be processed are found through a dynamic semantic difference set, which can be: calculating the peak cutting points under a preset threshold through the distance between the embedding vectors corresponding to each adjacent window in the dynamic semantic difference set; based on the peak decreasing strategy, the threshold is continuously lowered and the peak cutting points under the threshold are calculated until the peak cutting points are within the preset segmentation range; this can more accurately reflect the degree of semantic change between different windows in the text, and then the positions where the distance between the embedding vectors corresponding to each adjacent window exceeds the threshold are marked as semantic breakpoints, and the conclusions drawn will be more credible.

[0040] In one example, the number of peak cutting points under a preset threshold is calculated by the distance between the embedding vectors corresponding to each adjacent window in the dynamic semantic difference set, including: plotting the change relationship of the distance between the embedding vectors corresponding to each adjacent window in the dynamic semantic difference set as a line graph; and calculating the number of peak cutting points under the preset threshold according to the peak value shown in the line graph. Specifically: the change in the distance between the embedding vectors corresponding to each adjacent window is obtained by recording the distance change of each comparison during the above-mentioned comparison of the embedding vectors of the adjacent windows. These distance change values ​​are plotted into a line graph, which will show spikes (peaks) and troughs (valleys), reflecting the dynamic changes in the distance of the embedding vectors. These distance change value sets can be considered as a set of dynamic semantic differences. These value sets reflect the degree of semantic change between different windows in the text.

[0041] For example: a sharding range (Min-Max) can be set to guide the sharding threshold setting. Assume that the initial threshold is set to 100, and then the threshold is gradually lowered using a peak decreasing strategy. The dynamic semantic difference set is processed to calculate the peak number of cutting points under the current threshold. The calculated number of cutting points is compared with the set sharding range (Min-Max) for judgment. If the calculated number of cutting points hits (i.e. falls within) the set sharding range, the sharding is considered successful and the sharding operation is completed. If it does not hit, continue to lower the threshold and repeat the above steps until the calculated number of cutting points hits the set sharding range. The accuracy of the calculation results can be further enhanced by repeating the calculation and repeatedly verifying until the calculation results meet the requirements.

[0042] In step 103, the computer device combines the preliminary semantic segmentation and the semantic breakpoints to obtain the final semantic segmentation of the text to be processed.

[0043] Among them, the preliminary semantic segmentation results formed by hierarchical clustering and position reward and the breakpoint results found by the window mechanism do not conflict with each other. The two can refer to each other or further segment the text to be processed based on the results of one of them after the other is completed. That is, the preliminary semantic segmentation and semantic breakpoints can be combined to segment the file to be processed to obtain a more accurate segmentation result.

[0044] In addition, further clustering can be performed by comparing the preliminary semantic segmentation results and semantic breakpoints, and further clustering of different text clusters can be achieved by calculating the similarity between different text clusters. For example, on the preliminary semantic segmentation results formed based on hierarchical clustering and position rewards, the breakpoint results found by the window mechanism are used to further classify the preliminary semantic segmentation results. This method does not rely on predefined rules and can flexibly adapt to different types of text and complex semantic structures, and has broad application prospects.

[0045] In this embodiment, the sentences in the text to be processed are clustered by applying a hierarchical clustering algorithm and a position reward mechanism to form a preliminary semantic segmentation; the semantic breakpoints of the text to be processed are found by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; the final semantic segmentation of the text to be processed is obtained by combining the preliminary semantic segmentation and the semantic breakpoints. This effectively avoids the problem of semantic information loss caused by simple rule segmentation and greatly improves the accuracy and robustness of the segmentation.

[0046] The steps of the above method are divided only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0047] Another embodiment of the present invention relates to a semantic segmentation device, such as Figure 3 As shown, including:

[0048] Application clustering module 201, used to cluster sentences in the text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic fragments;

[0049] A semantic breakpoint module 202, configured to find the semantic breakpoints of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed;

[0050] The semantic processing module 203 is used to combine the preliminary semantic segmentation and the semantic breakpoints to obtain the final semantic segmentation of the text to be processed.

[0051] In one example, sentences in a text to be processed are clustered by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic fragmentation, including: converting the sentences in the text to be processed into corresponding high-dimensional vector representations; adding similarity scores to the high-dimensional vector representations corresponding to adjacent sentences in the text to be processed by the position reward mechanism in applying the hierarchical clustering algorithm to form preliminary semantic fragmentation; wherein the similarity scores are used to increase the probability that adjacent sentences in the text to be processed are clustered into the same cluster during the process of clustering the high-dimensional vector representations by applying the hierarchical clustering algorithm.

[0052] In one example, converting sentences in a text to be processed into corresponding high-dimensional vector representations includes: using a pre-trained text embedding model to convert each sentence in the text to be processed into a set of high-dimensional vector representations.

[0053] In one example, the semantic breakpoints of the text to be processed are found by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed, including: combining the sentences in the text to be processed to form multiple windows, each window containing one or more sentences; obtaining a dynamic semantic difference set of the text to be processed by calculating and comparing the embedding vectors corresponding to each window; and finding the semantic breakpoints of the text to be processed through the dynamic semantic difference set.

[0054] In one example, a dynamic semantic difference set of the text to be processed is obtained by calculating and comparing the embedding vectors corresponding to each window, including: respectively calculating the embedding vector corresponding to each window; comparing the distances between the embedding vectors corresponding to each adjacent window in turn by comparing them pairwise to obtain the dynamic semantic difference set.

[0055] In one example, the embedding vectors corresponding to each window are calculated respectively, including: performing embedding calculations on the sentences contained in the window in turn to obtain high-dimensional vector representations of the sentences contained in the window; and combining the high-dimensional vector representations of the sentences contained in the same window through a matrix algorithm to obtain the embedding vector corresponding to the window.

[0056] In one example, a dynamic semantic difference set is used to find semantic breakpoints of a text to be processed, including: calculating the peak cutting point number under a preset threshold value through the distance between the embedding vectors corresponding to each adjacent window in the dynamic semantic difference set; based on a peak decreasing strategy, continuously lowering the threshold and calculating the peak cutting point number under the threshold until the peak cutting point number is within a preset segmentation range; marking the position where the distance between the embedding vectors corresponding to each adjacent window exceeds the threshold as a semantic breakpoint.

[0057] In one example, the number of peak cutting points under a preset threshold is calculated by using the distance between the embedding vectors corresponding to each adjacent window in the dynamic semantic difference set, including: plotting the changing relationship of the distance between the embedding vectors corresponding to each adjacent window in the dynamic semantic difference set as a line graph; and calculating the number of peak cutting points under the preset threshold based on the peak value shown in the line graph.

[0058] In one example, the device also includes: a text preprocessing module, which is used to identify periods in the text to be processed and segment the text content of the text to be processed at the period position before clustering the sentences in the text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism to obtain sentences in each text to be processed.

[0059] In this embodiment, the sentences in the text to be processed are clustered by applying a hierarchical clustering algorithm and a position reward mechanism to form a preliminary semantic segmentation; the semantic breakpoints of the text to be processed are found by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; the final semantic segmentation of the text to be processed is obtained by combining the preliminary semantic segmentation and the semantic breakpoints. This effectively avoids the problem of semantic information loss caused by simple rule segmentation and greatly improves the accuracy and robustness of the segmentation.

[0060] It is not difficult to find that this embodiment is a device embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details mentioned in the above method embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above method embodiment.

[0061] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0062] Another embodiment of the present invention relates to an electronic device, such as Figure 4 As shown, it includes at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute the semantic segmentation method as described above.

[0063] The memory 302 and the processor 301 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 301 and the memory 302 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 301 is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor 301.

[0064] The processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 302 can be used to store data used by the processor 301 when performing operations.

[0065] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0066] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0067] Those skilled in the art will appreciate that the above-mentioned embodiments are specific examples for implementing the present invention, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A semantic sharding method, characterized in that: include: By applying the hierarchical clustering algorithm and the position reward mechanism, the sentences in the processed text are clustered to form preliminary semantic fragmentation; Finding semantic breakpoints of the text to be processed by comparing distances between embedding vectors corresponding to sentences in the text to be processed; The preliminary semantic segmentation and the semantic breakpoints are combined to obtain the final semantic segmentation of the text to be processed.

2. The semantic segmentation method according to claim 1, characterized in that: The method of clustering sentences in the text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic fragmentation includes: Converting sentences in the text to be processed into corresponding high-dimensional vector representations; In the applied hierarchical clustering algorithm, similarity scores are added to high-dimensional vector representations corresponding to adjacent sentences in the text to be processed by the position reward mechanism to form the preliminary semantic fragmentation; The similarity score is used to increase the probability that adjacent sentences in the text to be processed are clustered into the same cluster during the process of clustering the high-dimensional vector representation by applying the hierarchical clustering algorithm.

3. The semantic segmentation method according to claim 2, characterized in that: The step of converting the sentences in the to-be-processed text into corresponding high-dimensional vector representations includes: A pre-trained text embedding model is used to convert each sentence in the text to be processed into a set of high-dimensional vector representations.

4. The semantic segmentation method according to claim 1, characterized in that: The step of finding the semantic breakpoint of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed comprises: By combining sentences in the text to be processed to form a plurality of windows, each of the windows contains one or more sentences; By calculating and comparing the embedding vectors corresponding to each of the windows, a dynamic semantic difference set of the text to be processed is obtained; The semantic breakpoints of the text to be processed are found through the dynamic semantic difference set.

5. The semantic segmentation method according to claim 4, characterized in that: The step of obtaining a dynamic semantic difference set of the text to be processed by calculating and comparing the embedding vectors corresponding to each of the windows includes: Calculate the embedding vectors corresponding to each of the windows respectively; By comparing each pair of the windows with each other, the distances between the embedding vectors corresponding to the adjacent windows are compared in sequence to obtain the dynamic semantic difference set.

6. The semantic segmentation method according to claim 5, characterized in that: The respectively calculating the embedding vectors corresponding to the windows includes: Performing embedding calculations on the sentences contained in the window in sequence to obtain high-dimensional vector representations of the sentences contained in the window; The high-dimensional vector representations of the sentences contained in the same window are combined and calculated by a matrix algorithm to obtain an embedding vector corresponding to the window.

7. The semantic segmentation method according to claim 5, characterized in that: The step of finding the semantic breakpoint of the text to be processed through the dynamic semantic difference set includes: Calculating the number of peak cutting points under a preset threshold value by using the distance between the embedding vectors corresponding to each of the adjacent windows in the dynamic semantic difference set; Based on the peak decreasing strategy, the threshold is continuously lowered and the number of peak cutting points under the threshold is calculated until the number of peak cutting points is within the preset slicing range; Positions where the distances between the embedding vectors corresponding to the adjacent windows exceed the threshold are marked as semantic breakpoints.

8. The semantic segmentation method according to claim 7, characterized in that: The step of calculating the number of peak cutting points under a preset threshold by using the distances between the embedding vectors corresponding to the adjacent windows in the dynamic semantic difference set includes: Plotting the change relationship of the distance between the embedding vectors corresponding to each of the adjacent windows in the dynamic semantic difference set as a line graph; According to the peak value shown in the line graph, the number of peak cutting points under a preset threshold is calculated.

9. The semantic segmentation method according to claim 1, characterized in that: The method further comprises: Before clustering the sentences in the text to be processed by applying a hierarchical clustering algorithm and a position reward mechanism, the sentences in the text to be processed are obtained by identifying the periods in the text to be processed and segmenting the text content of the text to be processed at the positions of the periods.

10. A semantic slicing device, characterized in that: include: An application clustering module is used to cluster sentences in the processed text by applying a hierarchical clustering algorithm and a position reward mechanism to form preliminary semantic fragmentation; A semantic breakpoint module, used to find the semantic breakpoints of the text to be processed by comparing the distances between the embedding vectors corresponding to the sentences in the text to be processed; The semantic processing module is used to combine the preliminary semantic segmentation and the semantic breakpoints to obtain the final semantic segmentation of the text to be processed.

11. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the semantic segmentation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the semantic segmentation method according to any one of claims 1 to 9 is implemented.

Images