Text segmentation method and device, equipment, storage medium and product

Abstract

The invention discloses a text segmentation method and device, equipment, a storage medium and a product, and relates to the technical field of text processing, and the method comprises the steps that a target long text is acquired, and the target long text comprises a plurality of short sentences arranged in sequence; calculating a similarity score between the adjacent short sentences to obtain a similarity sequence of the target long text; smoothing the similarity sequence to obtain a smooth similarity sequence; and based on the smooth similarity sequence and the splitting threshold, segmenting the target long text. According to the method, the similarity score sequence is smoothed, so that the influence of extreme similarity scores on the whole sequence is weakened, misjudgment or errors caused by individual abnormal values are avoided, and the robustness of a text segmentation result is improved.

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a text segmentation method, device, equipment, storage medium and product. Background Art

[0002] In the related art, segmenting a long document is a basic and important step. The purpose of document segmentation is to divide a long document into smaller and easier to process units, which can be sentences, paragraphs or other semantic units.

[0003] However, the current document segmentation process may result in poor accuracy of the segmentation results due to influences such as noise, and the overall performance may be degraded due to individual outliers.

[0004] Therefore, how to improve the accuracy of text segmentation is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a text segmentation method, device, equipment, storage medium and product, aiming to solve the technical problem of how to improve the accuracy of text segmentation.

[0006] To achieve the above purpose, the present application proposes a text segmentation method, which includes: Acquire a target long text, wherein the target long text includes a plurality of short sentences arranged in sequence; Calculating similarity scores between adjacent short sentences to obtain a similarity sequence of the target long text; Performing smoothing processing on the similarity sequence to obtain a smoothed similarity sequence; The target long text is segmented based on the smoothed similarity sequence and the splitting threshold.

[0007] In some embodiments, smoothing the similarity sequence to obtain a smoothed similarity sequence includes: For a first similarity score in the similarity sequence, a smoothed similarity score corresponding to the first similarity score is obtained based on the first similarity score and the next similarity score of the first similarity value; For the last similarity score in the similarity sequence, based on the last similarity score and the previous similarity score of the last similarity value, a smoothed similarity score corresponding to the last similarity score is obtained; For any other similarity score except the first similarity score and the last similarity score in the similarity sequence, a smoothed similarity score corresponding to any other similarity score is obtained based on the other similarity score, the next similarity score of the other similarity score and the previous similarity score.

[0008] In some embodiments, segmenting the target long text based on the smoothed similarity sequence and the splitting threshold includes: Determine a preliminary segmentation score from the smoothed similarity sequence; Rearrange the smoothed similarities according to their numerical values ​​to obtain a new sequence; Determine the interquartile range value for the new series; Based on the interquartile range value, the split threshold is determined; When the preliminary segmentation score is less than or equal to the split threshold, the preliminary segmentation score is used as the effective segmentation score; The short sentences corresponding to the effective segmentation scores are used as topic boundary sentences to segment the target long text.

[0009] In some embodiments, determining the interquartile range value of the new sequence comprises: Determine the first quartile and the third quartile of the new series; For any one of the first quartile value and the third quartile value, determine the corresponding first parameter value and second parameter value in the new sequence, the sequence number value of the first parameter value in the new sequence is equal to the integer part of any one of the first quartile values, and the second parameter value is the next parameter value of the first parameter value; For any one of the first quartile value and the third quartile value, use the decimal part of any one of them as the second weight coefficient of the second parameter value, and determine the first weight coefficient of the first parameter value based on the second weight coefficient; For any one of the first quartile value and the third quartile value, based on the sum of the product of the first weight coefficient and the first parameter value and the product of the second weight coefficient and the second parameter value, update any one of them to determine the interquartile range value of the new sequence.

[0010] In some embodiments, determining a preliminary segmentation score from the smoothed similarity sequence includes: For each smoothed similarity score in the smoothed similarity sequence, when the smoothed similarity score is smaller than two adjacent smoothed similarity scores, the smoothed similarity score is determined as the preliminary segmentation score.

[0011] In some embodiments, the topic demarcation sentence is the ending sentence of the topic paragraph.

[0012] In addition, to achieve the above purpose, the present application also proposes a text segmentation device, which includes: A text acquisition module is used to acquire a target long text, where the target long text includes a plurality of short sentences arranged in sequence; The sequence acquisition module is used to calculate the similarity scores between adjacent short sentences and obtain the similarity sequence of the target long text; A smoothing processing module is used to perform smoothing on the similarity sequence to obtain a smoothed similarity sequence; The text segmentation module is used to segment the target long text based on the smooth similarity sequence and the splitting threshold.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a text segmentation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text segmentation method described above.

[0014] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text segmentation method described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the text segmentation method described above are implemented.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: By smoothing the similarity score sequence, the influence of extreme similarity scores on the overall sequence is reduced, misjudgment or error caused by individual outliers is avoided, and the robustness of the text segmentation results is improved. In addition, smoothing can also make the similarity score better reflect the contextual relationship between short sentences in the text, which helps to better understand the text structure and content, and improve the processing effect when performing natural language processing tasks on the target long text. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a text segmentation method provided by an embodiment of the present application is shown; Figure 2 A similarity sequence line graph before smoothing provided by an exemplary embodiment of the present application is shown; Figure 3A smoothed similarity sequence line graph after smoothing provided by an exemplary embodiment of the present application is shown; Figure 4 A schematic diagram of the structure of a text segmentation device provided in an embodiment of the present application is shown; Figure 5 A structural schematic diagram of a text segmentation device provided in an embodiment of the present application is shown.

[0020] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: obtaining a target long text, the target long text includes a plurality of short sentences arranged in sequence; calculating the similarity scores between adjacent short sentences to obtain a similarity sequence of the target long text; smoothing the similarity sequence to obtain a smoothed similarity sequence; and segmenting the target long text based on the smoothed similarity sequence and a segmentation threshold.

[0024] In the related technology, long text segmentation can be achieved by establishing entity information templates, semantically segmenting the long text to obtain text blocks, classifying the text blocks, predicting the entity information templates to which they belong, and sequence labeling the classified text blocks, extracting entity information to form structured data, and merging the structured data of multiple text blocks to obtain the structured results of the original insurance text, thereby achieving text segmentation, simplifying the content of the text, and facilitating quick reading of the text.

[0025] However, the segmentation method mentioned above uses a fixed threshold to obtain the segmentation position of the text, without considering whether the segmentation position corresponds to the change in data distribution. It is difficult to perform adaptive segmentation based on different topics of the text, which limits the generalization ability of text segmentation and leads to poor segmentation accuracy.

[0026] In summary, how to improve the accuracy of text segmentation is a problem that needs to be solved urgently.

[0027] Based on this, the present application provides a solution, which can reduce the impact of noise (such as individual outliers) on the segmentation position by smoothing the similarity sequence, thereby avoiding the overall segmentation performance degradation due to individual abnormal similarity scores and improving the accuracy of text segmentation.

[0028] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a text segmentation device capable of realizing the above functions, etc. The text segmentation device is taken as an example to illustrate this embodiment and the following embodiments.

[0029] Reference Figure 1 , Figure 1 The flowchart of the text segmentation method provided by an embodiment of the present application is shown. The text segmentation method can be applied to a text segmentation device, and includes the following steps S110 to S140: Step S110, obtaining a target long text, where the target long text includes a plurality of short sentences arranged in sequence.

[0030] The target long text refers to the text content extracted from the document file to be segmented. The document file to be segmented can be a file in a format such as PDF or Word, which is not limited in the present embodiment. For PDF files, the PDFMiner library can be used to extract the text content therein, and for Word documents, the python-docx library can be used to extract the text content.

[0031] The text content refers to the pure text part, which does not include non-text data formats such as tables and pictures. The text content extracted by the above library is combined to form the target long text according to its original arrangement order in the document.

[0032] In some implementations, the target long text can be processed into sentences according to preset rules. Specifically, the target long text can be first processed into sentences according to common sentence punctuation marks (such as a period ".", a question mark "?", etc.); then, according to regular expressions, characters such as spaces or line breaks in the document can be identified. It can be understood that in regular expressions, "\s" is a universal matcher used to match any blank character, and "\n" is used to match line breaks. The target long text is processed into sentences again through regular expressions, so that the target long text is divided into sentences to obtain multiple short sentences arranged in sequence in this embodiment.

[0033] Step S120, calculating similarity scores between adjacent short sentences to obtain a similarity sequence of the target long text.

[0034] The similarity score of a sentence is a quantitative indicator that measures the semantic similarity between two sentences. Through the similarity score, more similar short sentences can be classified into the same topic, thereby achieving more accurate segmentation of the target long text.

[0035] In some embodiments, in order to accurately capture the semantic relationship between short sentences, each short sentence can be converted into a vector representation. Among them, converting a short sentence into a vector representation can include a variety of embedding methods, which are not limited in the embodiments of the present application. As an example, a pre-trained bidirectional encoder converter can be used to represent the BERT model, that is, a short sentence is input into the BERT model, and the vector output by BERT is used as the vector representation of the short sentence. After obtaining the vector representation corresponding to each short sentence, the corresponding similarity score can be calculated for each short sentence.

[0036] Specifically, the vector representations corresponding to each short sentence can be arranged in the original order to obtain an embedded vector set. It can be understood that in the vector space, the similarity between two vectors can be measured by dot product distance and cosine similarity.

[0037] The following is an example of cosine similarity. In some implementations, for each short sentence in the embedding vector set, the similarity score can be calculated according to the current short sentence and the next short sentence using the following formula: sim_cos(S_E1,S_E2) Among them, S_E1 refers to the vector representation of the current short sentence, and S_E2 refers to the vector representation of the next short sentence.

[0038] In some implementations, for the last short sentence in the embedding vector set, the first short sentence in the embedding vector set may be defined as the next short sentence of the last short sentence.

[0039] For example, assuming there are five short sentences S1-S5 arranged in sequence, the similarity score of short sentence S1 can be calculated from the vector representation of short sentence S1 and short sentence S2, and so on; for short sentence S5, S1 can be defined as the "next" short sentence of S5, that is, the similarity score of short sentence S5 can be calculated from the vector representation of short sentence S5 and short sentence S1. In this way, the cosine similarity score corresponding to each short sentence in the embedded vector set can be calculated.

[0040] The cosine similarity score corresponding to each short sentence in the embedding vector set may be arranged in the order of arrangement of the short sentences in the embedding vector set to obtain the similarity sequence in step S120 of this embodiment.

[0041] It is understandable that, as mentioned above, the similarity sequence in step S120 is susceptible to large fluctuations due to certain irrelevant factors (ie, noise). Therefore, in order to reduce the impact of noise, the embodiment of the present application is designed to smooth the similarity sequence through step S130.

[0042] Step S130, smoothing the similarity sequence to obtain a smoothed similarity sequence.

[0043] In some embodiments, for the first similarity score in the similarity sequence, a smoothed similarity score corresponding to the first similarity score is obtained based on the first similarity score and the next similarity score of the first similarity value. For the last similarity score in the similarity sequence, a smoothed similarity score corresponding to the last similarity score is obtained based on the last similarity score and the previous similarity score of the last similarity value. For any other similarity score in the similarity sequence except the first similarity score and the last similarity score, a smoothed similarity score corresponding to any other similarity score is obtained based on any other similarity score, the next similarity score of any other similarity score and the previous similarity score.

[0044] It can be understood that the smoothing process is calculated based on the cosine similarity scores of adjacent short sentences of the short sentence to be smoothed, thereby reducing the fluctuation degree of adjacent cosine similarities.

[0045] Still taking the short sentences S1 to S5 in the above example as an example: Sentence S1 is only adjacent to S2, so the smoothed similarity score of sentence S1 is calculated by taking the average of S1 and S2.

[0046] For short sentences S2-S4, there are cosine similarity scores corresponding to adjacent short sentences, so the smoothed similarity score of the current short sentence can be obtained by calculating the cosine similarity score of the current short sentence and the average of the adjacent cosine similarity scores (that is, the average of the three).

[0047] Sentence S5 is only adjacent to S4, so the smoothed similarity score of sentence S5 is calculated by taking the average of S4 and S5.

[0048] In this embodiment, since the smoothed similarity score of each short sentence depends not only on itself but also on the adjacent sentences, the final smoothed similarity sequence is smoother and more continuous. Figure 2 and Figure 3 As shown, Figure 2 is the similarity score sequence before smoothing, Figure 3 This is the similarity score sequence after smoothing. Obviously, before smoothing, there are many noise points in the similarity score sequence. After smoothing, the noise points in the similarity score sequence can be reduced.

[0049] It is understandable that smoothing can make the similarity score better reflect the contextual relationship between short sentences, which helps to better understand the text structure and content. At the same time, it can also reduce the impact of extreme similarity scores (for example, a similarity score value is abnormally high / low) on the overall sequence, avoid misjudgment or error caused by individual outliers, and improve the robustness of the results.

[0050] Step S140 , segmenting the target long text based on the smoothed similarity sequence and the splitting threshold.

[0051] In some implementations, a preliminary segmentation score may be first determined from the smoothed similarity sequence, and then a final effective segmentation score may be determined from the preliminary segmentation score based on a split threshold, and finally the original target long text may be segmented based on the effective segmentation score.

[0052] It is understandable that when the smoothed similarity score of a short sentence is low, it reflects that the topics between adjacent short sentences have changed, that is, text segmentation may be required at the short sentence.

[0053] Therefore, in some implementations, for each smoothed similarity score in the smoothed similarity sequence, if the smoothed similarity score is smaller than the two adjacent smoothed similarity scores, the smoothed similarity score is determined as the preliminary segmentation score.

[0054] For example, assuming the smoothed similarity sequence is [0.805, 0.753, 0.71, 0.815, 0.8], then 0.71 is a preliminary segmentation score.

[0055] After obtaining all the preliminary segmentation scores in the smoothed similarity sequence, in some implementations, the smoothed similarities may be reordered according to numerical values ​​to obtain a new sequence; the interquartile range value of the new sequence may be determined; and the splitting threshold may be determined based on the interquartile range value.

[0056] It is understandable that the split threshold is used to screen the preliminary split scores again to ensure the rationality of the final effective split scores. The interquartile range (IQR) is often used to measure the degree of dispersion of data and can be used to remove outliers in the data set. By removing the "outliers" in the preliminary split scores, the effective split scores can be screened out.

[0057] For example, suppose there is a set of smoothed similarity sequences [0.531, 0.805, 0.753, 0.71, 0.815, 0.8, 0.82]. After rearranging them in ascending order, the new sequence obtained is [0.531, 0.71, 0.753, 0.8, 0.805, 0.815, 0.82].

[0058] In some embodiments, the first quartile value and the third quartile value of the new sequence may be obtained, and the interquartile range value IQR of the new sequence may be calculated using the first quartile value Q1 and the third quartile value Q3. Specifically, the first quartile value Q1 is the smoothed similarity score at the 25% position in the new sequence, and the third quartile value Q3 is the smoothed similarity score at the 75% position in the new sequence.

[0059] Q1 and Q3 can be calculated respectively by the following expressions: Pos(Q1)=(n+1) / 4; Pos(Q3)=3*(n+1) / 4; Where n is the number of smoothed similarity scores in the new sequence.

[0060] For Q1 or Q3, after the Pos value is calculated by the above expression, there are two cases: ① The Pos value is an integer; ② The Pos value is not an integer.

[0061] For case ①, the Pos-th smoothed similarity score in the new sequence can be directly obtained as Q1 / Q3.

[0062] For case ②, Q1 / Q3 can be calculated by interpolation.

[0063] In some embodiments, for any one of the first quartile value and the third quartile value, the corresponding first parameter value and second parameter value are determined in the new sequence, the sequence number value of the first parameter value in the new sequence is equal to the integer part of any one, and the second parameter value is the next parameter value of the first parameter value. For any one of the first quartile value and the third quartile value, the decimal part of any one is used as the second weight coefficient of the second parameter value, and the first weight coefficient of the first parameter value is determined based on the second weight coefficient. For any one of the first quartile value and the third quartile value, based on the sum of the product of the first weight coefficient and the first parameter value and the product of the second weight coefficient and the second parameter value, any one is updated.

[0064] Specifically, Q1 / Q3 can be calculated by the following expression:

[0065]

[0066] Where k is the integer part of Pos, that is, the sequence number of the first parameter value in the new sequence; is the smoothed similarity score at position k in the new sequence, i.e., the first parameter value; is the smoothed similarity score at position k+1 in the new sequence, that is, the second parameter value; f is the decimal part of Pos, that is, the second weight coefficient of the second parameter value. It can be understood that 1-f is the first weight coefficient.

[0067] For example, when Pos (Q1) is 3.5, based on the sequence [0.531, 0.71, 0.753, 0.8, 0.805, 0.815, 0.82], the calculation formula of Q1 is: Q1 = (1-0.5) * 0.753 + 0.5 * 0.8 = 0.7765.

[0068] After Q1 and Q3 are calculated by the above implementation, the difference between Q3 and Q1 can be used as IQR, that is, the interquartile range value is the difference between the third quartile value Q3 and the first quartile value Q1.

[0069] It can be understood that IQR reflects the degree of dispersion of the middle 50% of the data in the data set (new sequence). This degree of dispersion varies based on the changes in the subject of the target long text itself and is a relative probability reflection of the subject distribution of the target long text.

[0070] Specifically, the relationship between the interquartile range and the degree of dispersion is as follows: High dispersion: When the IQR value is large, it means that the range of the middle 50% of the data in the data set (new sequence) is wide, that is, the distribution of the data (smoothed similarity scores) within this range is more dispersed. Therefore, the dispersion is high.

[0071] Low dispersion: When the IQR value is small, it means that the range of the middle 50% of the data in the data set (new sequence) is narrow, that is, the distribution of the data (smoothed similarity score) in this range is more concentrated. Therefore, the dispersion is low.

[0072] In some implementations, if the preliminary segmentation score is less than or equal to the split threshold, the preliminary segmentation score is determined as a valid segmentation score.

[0073] As mentioned above, the interquartile range (IQR) can be used to eliminate outliers in a data set. By eliminating the "outliers" in the preliminary cut scores, effective cut scores can be screened out.

[0074] Specifically, for the interquartile range value, any data in the data set that is lower than Q1-1.5 * IQR will be considered an outlier and needs to be removed.

[0075] Therefore, in this embodiment, the splitting threshold is Q1-1.5*IQR. For the multiple preliminary splitting scores obtained in the above embodiment, it can be determined whether they are valid splitting scores by comparing them with the splitting threshold.

[0076] Through secondary screening of the split threshold, segmentation positions that have no practical significance can be filtered out to avoid inaccurate topic segmentation.

[0077] After obtaining the effective segmentation score, the short sentences corresponding to the effective segmentation score can be used as topic boundary short sentences to segment the target long text.

[0078] In some embodiments, the topic demarcation short sentence is the ending sentence of the topic paragraph in which it is located; in other embodiments, the topic demarcation short sentence is the first sentence of the next topic paragraph of the topic paragraph in which it is located.

[0079] Taking the topic dividing sentence as the ending sentence of the topic paragraph as an example, assuming that the third sentence is determined as the dividing point, then the first sentence, the second sentence and the third sentence are regarded as the same topic paragraph.

[0080] This embodiment provides a text segmentation method, which smoothes the similarity score sequence so that the subsequent text segmentation steps are not affected by extreme outliers, thereby reducing the impact on the subsequent splitting threshold, ensuring a more accurate splitting threshold, and indirectly improving the accuracy of text segmentation; by smoothing the interquartile range IQR of the similarity score sequence, determining the splitting threshold can better adapt to changes in data distribution in the document, facilitate adapting to multi-topic text segmentation work, and can ensure higher accuracy. In addition, compared with the use of fixed thresholds for text segmentation in related technologies, this application uses IQR to determine the splitting threshold, reducing the need for manual parameter setting, reducing the labor cost of adjusting parameters, and improving the automation level of the text segmentation algorithm.

[0081] This application also provides a text segmentation device, please refer to Figure 4 , the text segmentation device 100 comprises: A text acquisition module 110 is used to acquire a target long text, wherein the target long text includes a plurality of short sentences arranged in sequence; A sequence acquisition module 120 is used to calculate similarity scores between adjacent short sentences to obtain a similarity sequence of the target long text; A smoothing processing module 130, used for performing smoothing processing on the similarity sequence to obtain a smoothed similarity sequence; The text segmentation module 140 is used to segment the target long text based on the smoothed similarity sequence and the splitting threshold.

[0082] The text segmentation device 100 provided by the present application adopts the text segmentation method in the above embodiment, which can solve the technical problem of how to improve the accuracy of text segmentation. Compared with the prior art, the beneficial effects of the text segmentation device 100 provided by the present application are the same as the beneficial effects of the text segmentation method provided by the above embodiment, and other technical features in the text segmentation device 100 are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0083] The present application provides a text segmentation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text segmentation method in the above-mentioned embodiment 1.

[0084] Reference below Figure 5 , which shows a schematic diagram of the structure of a text segmentation device suitable for implementing the embodiment of the present application. The text segmentation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The text segmentation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0085] like Figure 5As shown, the text segmentation device 200 may include a processing device 210 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 220 or a program loaded from a storage device 230 to a random access memory (RAM: Random Access Memory) 240. In RAM240, various programs and data required for the operation of the text segmentation device are also stored. The processing device 210, ROM220 and RAM240 are connected to each other through a bus 250. An input / output (I / O) interface 260 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 260: an input device 270 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 280 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 230 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 290. The communication device 290 can allow the text segmentation device to communicate with other devices wirelessly or by wire to exchange data. Although the text segmentation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0086] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from the storage device 230, or installed from the ROM 220. When the computer program is executed by the processing device 210, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0087] The text segmentation device provided by the present application adopts the text segmentation method in the above embodiment, which can solve the technical problem of how to improve the accuracy of text segmentation. Compared with the prior art, the beneficial effects of the text segmentation device provided by the present application are the same as the beneficial effects of the text segmentation method provided by the above embodiment, and other technical features in the text segmentation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0088] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0089] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0090] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the text segmentation method in the above-mentioned embodiment.

[0091] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0092] The computer-readable storage medium may be included in the text segmentation device; or may exist independently without being assembled into the text segmentation device.

[0093] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the text segmentation device, the text segmentation device can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operations of the present application. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0096] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned text segmentation method, and can solve the technical problem of how to improve the accuracy of text segmentation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the text segmentation method provided in the above-mentioned embodiment, and will not be elaborated here.

[0097] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned text segmentation method when executed by a processor.

[0098] The computer program product provided by this application can solve the technical problem of how to improve the accuracy of text segmentation. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the text segmentation method provided by the above embodiment, which will not be repeated here.

[0099] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A text segmentation method, characterized in that: The text segmentation method comprises: Acquire a target long text, wherein the target long text includes a plurality of short sentences arranged in sequence; Calculating similarity scores between adjacent short sentences to obtain a similarity sequence of the target long text; Performing smoothing processing on the similarity sequence to obtain a smoothed similarity sequence; The target long text is segmented based on the smoothed similarity sequence and the splitting threshold.

2. The text segmentation method according to claim 1, characterized in that: The step of smoothing the similarity sequence to obtain a smoothed similarity sequence includes: For a first similarity score in the similarity sequence, based on the first similarity score and a next similarity score of the first similarity value, obtaining a smoothed similarity score corresponding to the first similarity score; For the last similarity score in the similarity sequence, based on the last similarity score and the previous similarity score of the last similarity value, obtaining a smoothed similarity score corresponding to the last similarity score; For any other similarity score in the similarity sequence except the first similarity score and the last similarity score, a smoothed similarity score corresponding to any other similarity score is obtained based on the other similarity score, a next similarity score of the other similarity score and a previous similarity score.

3. The text segmentation method according to claim 1, characterized in that: The step of segmenting the target long text based on the smoothed similarity sequence and the splitting threshold comprises: Determining a preliminary segmentation score from the smoothed similarity sequence; Rearranging the smoothed similarities according to numerical values ​​to obtain a new sequence; Determining the interquartile range value of the new sequence; Based on the interquartile range value, determining the split threshold; When the preliminary segmentation score is less than or equal to the splitting threshold, the preliminary segmentation score is used as the effective segmentation score; The short sentences corresponding to the effective segmentation scores are used as topic boundary short sentences to segment the target long text.

4. The text segmentation method according to claim 3, characterized in that: Determining the interquartile range value of the new sequence includes: Determining the first quartile value and the third quartile value of the new sequence; For any one of the first quartile value and the third quartile value, determine a corresponding first parameter value and a second parameter value in the new sequence, the sequence number value of the first parameter value in the new sequence is equal to the integer part of any one of the first quartile value, and the second parameter value is a parameter value next to the first parameter value; For any one of the first quartile value and the third quartile value, use the decimal part of any one of the first quartile value as the second weight coefficient of the second parameter value, and determine the first weight coefficient of the first parameter value based on the second weight coefficient; For any one of the first quartile value and the third quartile value, update the any one based on the sum of the product of the first weight coefficient and the first parameter value and the product of the second weight coefficient and the second parameter value to determine the interquartile range value of the new sequence.

5. The text segmentation method according to claim 3, characterized in that: Determining a preliminary segmentation score from the smoothed similarity sequence includes: For each smoothed similarity score in the smoothed similarity sequence, when the smoothed similarity score is smaller than two adjacent smoothed similarity scores, the smoothed similarity score is determined as the preliminary segmentation score.

6. The text segmentation method according to claim 3, characterized in that: The topic boundary sentence is the ending sentence of the topic paragraph.

7. A text segmentation device, characterized in that: The text segmentation device comprises: A text acquisition module, used to acquire a target long text, wherein the target long text includes a plurality of short sentences arranged in sequence; A sequence acquisition module, used to calculate the similarity scores between adjacent short sentences to obtain a similarity sequence of the target long text; A smoothing processing module, used for performing smoothing processing on the similarity sequence to obtain a smoothed similarity sequence; The text segmentation module is used to segment the target long text based on the smoothed similarity sequence and the splitting threshold.

8. A text segmentation device, characterized in that: The text segmentation device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text segmentation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text segmentation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the text segmentation method according to any one of claims 1 to 6 are implemented.

Images