Text segmentation method and device and electronic equipment

Abstract

The invention provides a text segmentation method and device and electronic equipment, and relates to the technical field of natural language processing. According to the scheme, for a file of which the text cannot be directly read by a computer, each form line is obtained through OCR recognition or PDF character extraction, each form line is subjected to predetermined processing to mark that each form line is a segmented sentence or a whole sentence, and a return character or a line feed character at the tail of each form line marked as the segmented sentence is deleted to obtain a target text after paragraph recombination. And obtaining a text segmentation result from the input text segmentation device suitable for Chinese. According to the scheme, redundant return characters or line feed characters in an OCR recognition or PDF character extraction result can be automatically removed, so that content typesetting input into the text divider is accurate, and then the accuracy of text segmentation is improved; according to the scheme, manual deletion operation is not needed, and manpower and time cost is saved; the above predetermined processing is realized based on rules, the realization process is high in efficiency, the occupied resources are few, and the response speed can be significantly improved when the method is used for the intelligent question-answering robot.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text segmentation method, device and electronic device. Background Art

[0002] When building an intelligent question-answering robot based on a general large language model and a privatized knowledge base, it is usually necessary to perform preprocessing work such as text clarity and text segmentation on private documents during the knowledge base creation stage. The main purpose is to reduce noise, save resources, improve processing efficiency and the accuracy of question-answering results. It helps the large language model to understand and process user questions more accurately and efficiently, making the system more intelligent and flexible, thereby better meeting user needs.

[0003] Existing Chinese text segmentation technologies mainly include dictionary-based or rule-based methods, deep learning-based methods, semantic-based methods, etc. The dictionary-based method uses a pre-defined dictionary to match and segment text. This method usually needs to be combined with certain rules for fuzzy matching and ambiguity elimination. The rule-based word segmentation method relies on expert experience and linguistic knowledge. The deep learning-based method uses deep neural networks, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), convolutional neural networks (CNN), etc., to automatically learn the text segmentation model. This method has great advantages in processing complex language features. The semantic-based method segments text by understanding the semantic information at the word and sentence levels. This method usually needs to be combined with natural language processing technology.

[0004] Existing Chinese text segmentation methods have good processing effects on files such as txt and word that can be directly read by computers, but the processing effects are poor for files such as PDF and images that cannot be directly read by computers. Summary of the invention

[0005] The present invention provides a text segmentation method, device and electronic device to solve the problem that the existing Chinese text segmentation method has poor text processing effect on PDF, image and other files.

[0006] In order to solve the above technical problems, the first aspect of the present specification provides a text segmentation method, comprising: obtaining a target text, wherein the target text includes a plurality of form lines obtained by performing OCR recognition on an image or extracting text from a PDF file, and each form line is split by a carriage return character and / or a line feed character; performing predetermined processing on each form line in turn to mark each form line as a sentence break or a complete sentence; deleting the carriage return character or line feed character at the end of each form line marked as a sentence break to obtain a target text after paragraph reorganization; inputting the target text into a text segmenter suitable for Chinese to obtain a text segmentation result; wherein the predetermined processing includes: S1, judging S1: Determine whether the current form line is a title; S2: if the current form line is a title, mark the current form line as a complete sentence; S3: if the current form line is not a title, determine whether the end of the current form line is the end symbol of a natural sentence; S4: if the end of the current form line is the end symbol of a natural sentence, mark the current form line as a complete sentence; S5: if the end of the current form line is not the end symbol of a natural sentence, determine whether the current form line contains an intermediate connector, or the previous form line is marked as a sentence break; S6: if the current form line contains an intermediate connector, or the previous form line is marked as a sentence break, mark the current form line as a sentence break.

[0007] In some embodiments, the predetermined processing also includes: S7, if the end of the current form line is not the end symbol of the natural sentence, the current form line does not contain an intermediate connector, and the previous form line is not marked as a sentence break, then mark the current form line as a complete sentence.

[0008] In some embodiments, determining whether the current form line is a title includes: determining whether the first M characters of the current form line contain predetermined characters, and the total number of characters in the current form line is less than or equal to N; wherein M and N are predetermined integers, M≤N; the predetermined characters include at least one of the following: a period, a space, Arabic numerals, and Chinese numerals; if the judgment result is yes, determining that the current form line is a title.

[0009] In some embodiments, before determining whether the current formal line is a title, the process also includes: obtaining multiple sample texts whose formal lines do not conform to natural line typesetting rules; extracting each title from each sample text; counting the number of characters of each extracted title; calculating the mean and standard deviation of the number of characters of each title; determining a confidence interval corresponding to a predetermined confidence level; and determining the value of N based on the maximum value in the confidence interval.

[0010] In some embodiments, the process of performing OCR recognition or PDF text extraction on the original text also includes: identifying the font format of each form line; the font format includes at least one of the following: font type, font size, whether it is italic, whether it is bold; combining the font format of each form line to determine whether each form line is a title.

[0011] In some embodiments, whether each form line is a title is determined in combination with the font format of each form line, including: judging whether the first M characters of the current form line contain predetermined characters, and the font format of the current form line is different from the form lines of the predetermined proportion of the original text; the predetermined characters include at least one of the following: period, space, Arabic numerals, Chinese numerals; if the judgment result is yes, the current form line is determined to be a title.

[0012] In some embodiments, the value of N is determined in the following manner: determining the authoritative website associated with the original text corresponding to the target text; crawling the typesetting requirement file corresponding to the text from the authoritative website; and parsing the maximum number of characters of the title from the typesetting requirement file as the value of N.

[0013] In some embodiments, after the target text is input into a text segmenter applicable to Chinese and a text segmentation result is obtained, it also includes: obtaining the content of the user's question; comparing the similarity of each text segmentation result with the content of the user's question; screening out multiple segmentation results with the greatest similarity to the content of the user's question; generating prompt words based on the multiple segmentation results with the greatest similarity; inputting the prompt words into a question-and-answer big model, and the question-and-answer big model outputs a reply to the content of the user's question.

[0014] The second aspect of the present specification provides a text segmentation device, comprising: a first acquisition unit, used to acquire a target text, wherein the target text includes a plurality of formal lines obtained by performing OCR recognition on an image or extracting text from a PDF file, and each formal line is split by a carriage return character and / or a line feed character; a marking unit, used to perform predetermined processing on each formal line in turn to mark each formal line as a sentence break or a complete sentence; a reorganization unit, used to delete the carriage return character or line feed character at the end of each formal line marked as a sentence break to obtain a target text after paragraph reorganization; a segmentation unit, used to input the target text into a text segmenter suitable for Chinese to obtain a text segmentation result; wherein the predetermined processing is performed on each formal line in turn to mark each formal line as a sentence break or a complete sentence; The processing includes: S1, judging whether the current form line is a title; S2, if the current form line is a title, marking the current form line as a complete sentence; S3, if the current form line is not a title, judging whether the end of the current form line is the end symbol of a natural sentence; S4, if the end of the current form line is the end symbol of a natural sentence, marking the current form line as a complete sentence; S5, if the end of the current form line is not the end symbol of a natural sentence, judging whether the current form line contains an intermediate connector, or the previous form line is marked as a sentence break; S6, if the current form line contains an intermediate connector, or the previous form line is marked as a sentence break, marking the current form line as a sentence break.

[0015] In some embodiments, the device also includes: a second acquisition unit, used to acquire the content of the user's question; a comparison unit, used to compare the similarity of each text segmentation result with the content of the user's question; a screening unit, used to screen out multiple segmentation results with the greatest similarity to the content of the user's question; a generation unit, used to generate prompt words based on the multiple segmentation results with the greatest similarity; and a reply unit, used to input the prompt words into the question and answer big model, and the question and answer big model outputs a reply to the content of the user's question.

[0016] The third aspect of this specification provides an electronic device, comprising: a memory and a processor, the processor and the memory are communicatively connected to each other, the memory stores computer instructions, and the processor implements the text segmentation method described in any one of the first aspects by executing the computer instructions.

[0017] A fourth aspect of the present specification provides a computer storage medium, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the text segmentation method described in any one of the first aspects are implemented.

[0018] A fifth aspect of the present specification provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the text segmentation method described in any one of the first aspects.

[0019] The text segmentation method provided in this specification, for files whose text cannot be directly read by a computer, first obtains each form line through OCR recognition or PDF text extraction, performs predetermined processing on each form line to mark each form line as a sentence break or a complete sentence, deletes the carriage return or line feed characters at the end of each form line marked as a sentence break, obtains the target text after paragraph reorganization, and inputs the reorganized target text into a text segmenter suitable for Chinese to obtain the text segmentation result. This solution can automatically remove redundant carriage return or line feed characters in the results of OCR recognition or PDF text extraction, so that the content input to the text segmenter is accurately formatted, thereby improving the accuracy of text segmentation; this solution does not require manual deletion operations, saving manpower and time costs; the above predetermined processing is implemented based on rules, rather than based on network models, and the implementation process of the predetermined processing is highly efficient and occupies less resources, and can significantly improve the answering speed when used in intelligent question-answering robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 It is a schematic diagram of a file where the computer cannot directly read the text;

[0022] Figure 2 Yes Figure 1 A schematic diagram of the results obtained by directly inputting the OCR recognition or PDF extraction results of the file shown into the text segmenter;

[0023] Figure 3 yes Figure 1 A schematic diagram of a text segmentation result obtained by processing a file after OCR recognition using the text segmentation method provided in this specification;

[0024] Figure 4 It is a schematic diagram of the text segmentation method provided in this manual;

[0025] Figure 5 It is a schematic diagram of the predetermined treatment;

[0026] Figure 6 It is a schematic diagram of the results of OCR recognition or PDF text extraction;

[0027] Figure 7 yes Figure 6 The target text shown is in Figure 5 A schematic diagram of the result obtained after paragraph reorganization by the method shown;

[0028] Figure 8 A schematic diagram of an original file;

[0029] Fig. 9 A schematic diagram of a complete sentence without punctuation in the original document;

[0030] Fig.10 is another schematic diagram of the text segmentation method provided in this specification;

[0031] Fig.11 A schematic diagram of a text segmentation device provided in this specification;

[0032] Fig.12 Another schematic diagram of the text segmentation device provided in this specification;

[0033] Fig.13 This is a schematic diagram of the structure of the electronic device provided in this manual. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0035] It should also be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0036] The existing Chinese text segmentation method for processing PDF, image and other files that cannot be directly read by computers usually first performs OCR (Optical Character Recognition) on the image file to extract the text in the file, or extracts the text in the PDF file to obtain a text file that can be directly read by the computer, and then inputs the text file into the text segmenter. The inventors found that the reason why the text segmenter does not work well for such files is that the OCR recognition results or PDF text extraction results include a large number of redundant carriage return characters or line feed characters. Figure 1 For example, after recognizing and extracting text content through OCR technology, the recognized text still retains the layout in the picture, that is, the sentences are arranged from top to bottom, and there is a carriage return or line feed at the end of each line, so that consecutive sentences are also separated by carriage returns or line feeds. Figure 2 for Figure 1 The schematic diagram of the text segmentation result obtained after the file shown is subjected to OCR recognition and input into the text segmenter (ChineseRecursive), wherein the dotted line represents the separator.

[0037] The above problems also exist in the extraction results of PDF files. This is because the main purpose of PDF files is to ensure that the format and layout of the document remain consistent on any device. In PDF files, the text is precisely controlled according to page numbers and layouts, rather than simply processed as a continuous text stream. Therefore, when extracting the content of a PDF file, the original page layout and typesetting are usually retained, resulting in a line break at the end of each line. Each line in a PDF file is actually an independent text block or paragraph.

[0038] Based on this, this specification provides a text segmentation method, which removes redundant carriage returns or line breaks in OCR recognition and PDF extraction results by setting rules. Figure 3 for Figure 1 The document shown is processed by OCR recognition and the text segmentation method provided in this specification (using ChineseRecursive text segmenter) to obtain a schematic diagram of the text segmentation result, wherein the dotted line represents the separator.

[0039] Specifically, Figure 4 As shown, the text segmentation method includes the following steps:

[0040] S10: Obtain a target text, wherein the target text includes a plurality of formal lines obtained by performing OCR recognition on an image or extracting text from a PDF file, and each formal line is split by a carriage return character and / or a line feed character.

[0041] After the user asks a question to the intelligent question-answering robot with a general large language model and a private knowledge base, the intelligent question-answering robot will automatically process the knowledge in each file in the knowledge base and give an answer based on the processing results. The knowledge base is usually built based on a large number of existing files, a large number of which are files that cannot be directly read by computers, such as PDF and image format files. Although the text content in such files can be automatically processed by OCR recognition tools or PDF extraction tools, there is an additional carriage return or line feed at the end of each line in the processing result, so that consecutive sentences are also separated by carriage returns or line feeds.

[0042] In the prior art, text segmentation models applicable to Chinese usually refer to the carriage return or line feed characters in the text for segmentation. Therefore, redundant carriage return and line feed characters in the target text will greatly affect the accuracy of the segmentation results of the text segmentation model.

[0043] A form line refers to a line in an OCR recognition or PDF extraction result. Each form line usually ends with a carriage return character or a line feed character, that is, a form line is distinguished by a carriage return character or a line feed character in an OCR recognition or PDF extraction result.

[0044] S20: performing predetermined processing on each form line in turn to mark each form line as a segmented sentence or a complete sentence.

[0045] like Figure 5 As shown, the predetermined process includes the following S1 to S6.

[0046] S1, determine whether the current form line is a title.

[0047] S2, if the current form line is a title, mark the current form line as a whole sentence.

[0048] Normally, there is a carriage return or line break at the end of the title in an article. However, many carriage returns or line breaks are added in the OCR recognition or PDF extraction results, which are easily confused with the carriage return or line break at the end of the title, causing the text segmenter to be unable to correctly segment the text content.

[0049] To this end, you can first identify the title in the OCR recognition or PDF extraction results, and mark the formal line where the title is located as a complete sentence, so that when you delete the extra carriage return or line feed characters in the text later, the carriage return or line feed characters at the end of the formal line where the title is located can be retained.

[0050] S3, if the current form line is not a title, determine whether the end of the current form line is the end symbol of a natural sentence.

[0051] A natural sentence is a sentence with complete meaning. Natural sentences in a text are usually distinguished by the end symbol of the natural sentence, and one end symbol corresponds to one natural sentence. The end symbol of a natural sentence can be, for example: ".", "!", "?", "...".

[0052] S4, if the end of the current form line is the end symbol of a natural sentence, the current form line is marked as a complete sentence.

[0053] When the end of the current form line is the end symbol of a natural sentence, marking the current form line as a complete sentence can facilitate retaining the carriage return or line feed at the end of the current form line when deleting redundant carriage returns or line feeds in the text later.

[0054] Generally, when performing text segmentation, text segmenters focus more on complete sentences rather than paragraph divisions in the original text. For example, if the sentences A, B, C, D, and E are arranged in sequence and fit into one segmentation unit as a whole, whether these five sentences belong to one paragraph has almost no effect on the semantic segmentation results.

[0055] exist Figure 6 In the example, the ends of the first, fourth, sixth and tenth lines are the end symbols of natural sentences, and these formal lines can be marked as complete sentences.

[0056] S5, if the end of the current form line is not the end symbol of the natural sentence, it is determined whether the current form line contains an intermediate connector, or the previous form line is marked as a sentence break.

[0057] In the case where a format line is not a header and has no end symbol at the end, such as Figure 6 As shown, there may be three situations: 1. The current form line is a long sentence with no punctuation marks, see Figure 6③ in the line; 2. There are punctuation marks as middle connectors in the current form line. The punctuation marks as middle connectors can be, for example, ",","," and so on. See Figure 6 ②, ⑦ and ⑧ in the current form line; 3. The current form line includes the end symbols of the natural sentence ".", "!", "?", "...", etc., see Figure 6 4. The current form line contains both punctuation marks as intermediate connectors and the end of a natural sentence, see Figure 6 The ninth line in .

[0058] In the second case above, the previous form line may be a complete sentence, that is, the beginning of the current form line is the beginning of a natural sentence; the previous form line may also be a broken sentence, that is, the beginning of the current form line is connected to the end of the previous form line to form a complete sentence.

[0059] For the first and second cases, it is obvious that the current form line is a sentence break. For the third and fourth cases, since the OCR recognition or PDF extraction results arrange the content at both ends of the end symbol on a form line, the content at both ends of the end symbol must be highly correlated and located in a natural paragraph. In this case, the current form line can be marked as a sentence break so that the carriage return or line feed at the end of the current form line can be deleted later, so that the current form line can be spliced ​​with the next form line.

[0060] S6, if the current form line contains an intermediate connector, or the previous form line is marked as a sentence break, then mark the current form line as a sentence break.

[0061] When the end of the current form line is not the end symbol of a natural sentence, if the current form line contains an intermediate connector, it belongs to the second case mentioned above and should be marked as a sentence break.

[0062] When the end of the current formal line is not the end symbol of a natural sentence, if the previous formal line is marked as a sentence break, it may belong to the first case mentioned above and should also be marked as a sentence break.

[0063] When the end of the current form line is not the end symbol of a natural sentence, by judging whether the current form line contains an intermediate connector or whether the previous form line is marked as a sentence break, it can be determined whether the current form line is a sentence break, thereby facilitating the determination of whether to delete the carriage return or line feed character at the end of the current form line.

[0064] S7: If the end of the current form line is not the end symbol of a natural sentence, the current form line does not contain an intermediate connector, and the previous form line is not marked as a sentence break, then mark the current form line as a complete sentence.

[0065] S30: deleting the carriage return character or line feed character at the end of each line marked as a sentence break, to obtain a target text after paragraph reorganization.

[0066] Figure 6 The text shown is in Figure 5 After reorganizing the paragraphs using the method shown, we can get Figure 7 The paragraph layout shown.

[0067] It should be noted that, under normal circumstances, if the end of a formal line is not the end symbol of a natural sentence, it can be determined that the natural sentence is a broken sentence. However, in some embodiments, the original file contains a complete sentence without punctuation. The "complete sentence without punctuation" can specifically be at least one of the following: an emergency mark of the file, a confidentiality statement of the file, the signature of the sender or the sending unit of the file, the date of issuance of the file, etc. Figure 8 shown.

[0068] In the above step S7, when the end of the current form line is not the end symbol of a natural sentence, the current form line is not directly determined as a sentence break, but this situation is further divided into the following two situations: (1) the current form line contains an intermediate connector, or the previous form line is marked as a sentence break; (2) the current form line does not contain an intermediate connector, and the previous form line is not marked as a sentence break. The above situation (2) is for the above-mentioned "whole sentence without punctuation". In other words, the above S7 can be prepared to identify the "whole sentence without punctuation" in the original file, so that the target text can be accurately reorganized into paragraphs, so that the content layout of the input text segmenter is more accurate, thereby improving the accuracy of text segmentation.

[0069] In some cases, "complete sentences without punctuation" may also appear in the body of the original document. For example, the original document contains a long sentence that is longer than 2 lines, such as Fig. 9 In this case, the S7 will Fig. 9 The second and third lines in the text are marked as complete sentences, which results in a certain error rate in paragraph reorganization. In practice, the probability of such long sentences appearing is very low, so it usually does not have a significant impact on the accuracy of text segmentation.

[0070] In some embodiments, in order to further improve the accuracy of paragraph reorganization and text segmentation, before the above step S7, step S8 is further included: input the content of the previous form line and the current form line into a pre-trained network model, and determine whether the current form line is marked as a sentence break according to the result output by the network model. If the current form line is not marked as a sentence break, step S7 is executed to mark the current form line as a complete sentence.

[0071] The above network model can be trained in the following manner: extract the previous form line and the current form line of the following content from multiple sample files in advance as data for a training sample, and set the sample label as the first label: the emergency mark of the file, the confidentiality statement of the file, the signature of the sender or the sending unit of the file, and the date of issuance of the file; extract two consecutive lines without punctuation from the text of multiple sample files in advance as a training sample, and set the sample label as the second label; use the data of the above training samples as input data of the network model, and use the sample label as output data of the network model to train the network model. The network model trained in this way can correctly judge whether there is a correlation between the current form line and the previous form line, that is, whether the current form line can be marked as a sentence break. The above S8 can specifically determine that the current form line is marked as a sentence break when the network model output is the first label.

[0072] S40: Input the target text into a text segmenter suitable for Chinese to obtain a text segmentation result.

[0073] The text segmenter suitable for Chinese can adopt any text segmenter in the prior art. For example, ChineseRecursiveTextSplitter, ChineseTextSplitter and AliTextSplitter are three popular methods for Chinese text segmentation, which are widely used in the document processing of intelligent question-answering robots based on privatized knowledge bases. Among them, ChineseRecursiveTextSplitter is a recursive segmentation method that processes Chinese text by recursively splitting the text into smaller parts. This method is suitable for processing longer Chinese texts and splitting them into small pieces that are easier to process and analyze. ChineseTextSplitter is a character-based text segmentation method that segments Chinese through a series of characters. This method can handle punctuation and spaces in Chinese text and split them together with other characters. AliTextSplitter is a text segmentation method based on deep learning. It is a sequence model used to model document segmentation as a sentence-level sequence labeling task, and proposes an adaptive sliding window method to further improve the reasoning efficiency.

[0074] When performing text segmentation on files in a database, the prior art usually requires that the files in the database must be files that can be directly read by computers. For files that cannot be directly read by computers, some application scenarios do not process them; even if they are processed, they usually first perform OCR recognition on the files to obtain text files, and then input the text files into the text segmentation model.

[0075] In order to improve the accuracy of text segmentation, it is usually required to manually adjust the OCR recognition or PDF extraction results to a normal layout. However, in some scenarios, the number of database files to be processed is large, and manual adjustment of the layout consumes a lot of time and effort. Therefore, it is unrealistic to manually adjust the layout of OCR recognition or PDF extraction results.

[0076] The text segmentation method provided in this specification, for files whose text cannot be directly read by a computer, first uses OCR to identify each form line, performs predetermined processing on the OCR recognition or PDF extraction results to mark each form line as a sentence break or a complete sentence, deletes the carriage return or line feed characters at the end of each form line marked as a sentence break, obtains the target text after paragraph reorganization, and inputs the reorganized target text into a text segmenter suitable for Chinese to obtain the text segmentation result. This solution can automatically remove redundant carriage return or line feed characters in the OCR recognition or PDF extraction results of the text, so that the content input to the text segmenter is accurately formatted, thereby improving the accuracy of text segmentation; this solution does not require manual deletion operations, saving manpower and time costs; the above predetermined processing is implemented based on rules, rather than based on network models, and the implementation process of the predetermined processing is highly efficient and occupies less resources, and can significantly improve the answering speed when used in intelligent question-answering robots.

[0077] In some embodiments, the above S1 can specifically be: determine whether the first M characters of the current form line contain predetermined characters, and the total number of characters in the current form line is less than or equal to N; wherein M and N are predetermined integers, M≤N; the predetermined characters include at least one of the following: period, space, Arabic numerals, Chinese numerals; if the judgment result is yes, determine that the current form line is a title.

[0078] Usually, the subheadings in the article are arranged on the left, in the form of "Three, XXX", "1.2XXX", and the numbers "Three" and "1.2" are located at the beginning of a line. Therefore, there must be a period or space, Arabic numerals or Chinese numerals in the first few characters of the title line. The specific value of M can be one of 4, 5, 6, 7, and 8.

[0079] The title line is a highly refined summary of the content, so the length of the title is usually limited. The maximum value N of the title length can be pre-set, and only when the total number of characters in the form line is less than N, the form line is determined to be a title.

[0080] Before determining whether the current format row is a title, the value of N can be determined through the following S01 to S06.

[0081] S01: Acquire multiple sample texts whose formal lines do not conform to natural line typesetting rules.

[0082] Multiple files whose texts cannot be directly read by a computer can be extracted from a file database of the same type as the target text, and then recognized by OCR, and the OCR recognition or PDF extraction results can be used as corresponding sample files.

[0083] S02: Extract each title from each sample text, either manually or by other means.

[0084] S03: Count the number of characters of each extracted title.

[0085] S04: Calculate the average and standard deviation of the number of characters in each title.

[0086] S05: Determine a confidence interval corresponding to a predetermined confidence level.

[0087] S06: Determine the value of N according to the maximum value in the confidence interval.

[0088] For example, if the average value of the number of characters in each title is μ and the standard deviation is σ, then the 95% confidence interval is [μ-2σ, μ+2σ], where μ+2σ is the maximum value in the confidence interval, and the maximum integer less than or equal to or greater than or equal to μ+2σ can be used as the value of N. The value of the predetermined confidence level can be 95% or 90%.

[0089] By counting the number of characters of each title in the sample text, calculating the mean value, standard deviation, and then calculating the confidence interval, the value interval of the number of characters in the title of most texts can be determined. Then, determining the maximum value of the confidence interval as the value of N can ensure that the title judgment of most target texts is accurate. Inevitably, there are some titles with large character counts. If the maximum number of characters is used as the value of N, it may cause the sentences in the text to be misjudged as titles, and then cause sentence segmentation errors. It can be seen that S01 to S06 can not only ensure that the title judgment of most target texts is accurate, but also reduce the misjudgment of text sentences. The above statistical method determines the value of N, which is closer to the writing habits of the author of the article. Judging the value of N from the actual content of the article, and then judging whether the formal line is a title based on the value of N, will make the judgment result more accurate.

[0090] Before determining whether the current form line is a title, the value of N can also be determined by the following methods: determining the authoritative website associated with the original text corresponding to the target text; crawling the typesetting requirement file corresponding to the text from the authoritative website; parsing the maximum number of characters of the title from the typesetting requirement file as the value of N.

[0091] By crawling the typesetting requirement file from the authoritative website to determine the value of N, the value of N can be determined more accurately.

[0092] In some texts, the format of the title is different from that of the text. For example, the title is bold / italicized while the text is not bold / italicized, the font size of the title is larger than the text, the fonts used for the title and the text are different, etc. The difference in font format can distinguish the title from the text.

[0093] Based on this, in some other embodiments, the above S1 can also determine whether the form line is a title according to the font format. Specifically, the existing OCR recognition tool can be improved so that it can recognize the font format of each form line during the OCR recognition of the original text; the font format includes at least one of the following: font type, font size, whether it is italic, and whether it is bold. The above S1 can determine whether each form line is a title according to the improved OCR recognition or PDF extraction results combined with the font format of each form line.

[0094] Determine whether each form line is a title in combination with the font format of each form line, for example, by determining the form line corresponding to the text in the target font format as a title, wherein the proportion of form lines using the target font is less than or equal to a predetermined proportion, such as 20%, 10%, etc.

[0095] Determine whether each form line is a title in combination with the font format of each form line. For example, if the first M characters of the current form line include the predetermined character, and the total number of characters in the current form line is less than or equal to N, and the font format of the current form line is more prominent, then determine that the current form line is a title. The font format is more prominent, which means bold, italic, larger font size, etc. For example, if a text contains two font formats, the first font format is bold, and the second font format is not bold, and other aspects of the font formats are the same, then the first font format is more prominent than the second font format.

[0096] Determine whether each form line is a title in combination with the font format of each form line. For example, it can also be determined as follows: determine whether the first M characters of the current form line contain periods or spaces, and whether the font format of the current form line is different from the form lines of a predetermined proportion of the original text; if the judgment result is yes, determine that the current form line is a title.

[0097] The predetermined ratio may be greater than or equal to 80%, 90%, etc. That is, whether a form line is a title is determined based on whether the first M characters contain dots or spaces, and whether the font format is different from most form lines in the original text.

[0098] The above font format may be obtained synchronously during the process of OCR recognition or PDF text extraction.

[0099] In some embodiments, Fig.10 As shown, after the above S40, the following S50 to S90 are also included.

[0100] S50: Obtain the content of the user's question.

[0101] S60: Compare the similarity between each text segmentation result and the content of the user's question.

[0102] S70: Filter out multiple segmentation results that are most similar to the content of the user's question.

[0103] S80: Generate prompt words according to the multiple segmentation results with the greatest similarity.

[0104] S90: Input the prompt word into the question-answering model, and the question-answering model outputs a reply to the user's question.

[0105] Conversations with large language models (LLMs) are often referred to as "prompts" or "input prompts" because of how these models work and interact. LLMs (such as the GPT family) work by using deep learning models based on the Transformer architecture, which are trained to predict the next word or token given a sequence of text. To achieve this, the model needs some form of "start" or "prompt" in order to generate relevant and meaningful output.

[0106] When a user interacts with an LLM, they typically provide a short text snippet as input, which is called a “prompt word.” This prompt word can be a question, an instruction, a description, or any other form of text, and its purpose is to stimulate the model to generate a response relevant to the prompt.

[0107] This solution first determines the content in the database that is highly similar to the user's question, and then generates prompt words based on these contents, so that the prompt words are highly relevant to the content in the database, so that the content output by the question-answering model is also highly relevant to the content in the database, that is, it realizes the use of knowledge in the database to answer the user's questions.

[0108] The above-mentioned S10 to S40, and S50 to S90, can make it possible to accurately segment the text files in the database that cannot be directly read by the computer, so as to accurately match the content with a high similarity to the user's question content from such files, so as to make the generated prompt words more appropriate, and thus the answer content output by the question and answer model more accurate.

[0109] This specification provides a text segmentation device that can be used to implement the above-mentioned text segmentation method. Fig.11As shown, the device includes a first acquisition unit 10, a labeling unit 20, a recombination unit 30 and a segmentation unit 40.

[0110] The first acquisition unit 10 is used to acquire a target text, wherein the target text includes a plurality of formal lines obtained by performing OCR recognition on an image or extracting text from a PDF file, and each formal line is split by a carriage return character and / or a line feed character.

[0111] The marking unit 20 is used to perform predetermined processing on each form line in turn to mark each form line as a segmented sentence or a complete sentence.

[0112] The reorganization unit 30 is used to delete the carriage return character or line feed character at the end of each line marked as a sentence break, so as to obtain the target text after paragraph reorganization.

[0113] The segmentation unit 40 is used to input the target text into a text segmenter suitable for Chinese to obtain a text segmentation result.

[0114] The predetermined process includes the following steps S1 to S7.

[0115] S1, determine whether the current form line is a title;

[0116] S2, if the current form line is a title, mark the current form line as a complete sentence;

[0117] S3, if the current form line is not a title, determine whether the end of the current form line is the end symbol of a natural sentence;

[0118] S4, if the end of the current form line is the end symbol of a natural sentence, mark the current form line as a complete sentence;

[0119] S5, if the end of the current form line is not the end symbol of the natural sentence, then determine whether the current form line contains an intermediate connector, or the previous form line is marked as a sentence break;

[0120] S6, if the current form line contains an intermediate connector, or the previous form line is marked as a sentence break, then mark the current form line as a sentence break.

[0121] In some embodiments, Fig.12 As shown, the device further includes a second acquiring unit 50 , a comparing unit 60 , a screening unit 70 , a generating unit 80 and a replying unit 90 .

[0122] The second acquisition unit 50 is used to acquire the content of the question asked by the user.

[0123] The comparison unit 60 is used to compare the similarity between each text segmentation result and the content of the user's question.

[0124] The screening unit 70 is used to screen out a plurality of segmentation results that have the greatest similarity to the content of the user's question.

[0125] The generating unit 80 is used to generate a prompt word according to the plurality of segmentation results with the greatest similarity.

[0126] The answering unit 90 is used to input the prompt word into the question-answering model, and the question-answering model outputs a reply to the user's question content.

[0127] The description and functions of the above-mentioned devices can be understood by referring to the content of the text segmentation method part, which will not be repeated here.

[0128] The embodiment of the present invention further provides an electronic device, such as Fig.13 As shown, the electronic device may include a processor 1301 and a memory 1302, wherein the processor 1301 and the memory 1302 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0129] The processor 1301 may be a central processing unit (CPU). The processor 1301 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0130] The memory 1302 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the text segmentation method in the embodiment of the present invention (e.g. Fig.11 The processor 1301 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 1302, that is, the text segmentation method in the above method embodiment is implemented.

[0131] The memory 1302 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 1301, etc. In addition, the memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1302 may optionally include a memory remotely arranged relative to the processor 1301, and these remote memories may be connected to the processor 1301 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The one or more modules are stored in the memory 1302, and when executed by the processor 1301, the following is performed: Figure 4 and Figure 5 The text segmentation method in the illustrated embodiment.

[0133] The specific details of the above electronic device can be understood by referring to the corresponding descriptions and effects in the method embodiment, and will not be repeated here.

[0134] The present specification also provides a computer storage medium, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the above-mentioned text segmentation method are implemented.

[0135] The present specification also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned text segmentation method are implemented.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0137] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0138] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0139] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0140] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute certain parts of the methods of each implementation method of the present application.

[0141] The present application can be used in many general or special computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0142] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0143] Although the present application has been described through embodiments, those skilled in the art will appreciate that there are many modifications and variations to the present application without departing from the spirit of the present application, and it is intended that the appended claims include these modifications and variations without departing from the spirit of the present application.

Claims

1. A text segmentation method, characterized in that: include: Obtaining a target text, wherein the target text includes a plurality of formal lines obtained by performing OCR recognition on an image or extracting text from a PDF file, wherein each formal line is split by a carriage return character and / or a line feed character; Performing predetermined processing on each form line in turn to mark each form line as a broken sentence or a complete sentence; Delete the carriage return character or line feed character at the end of each line marked as a sentence break to obtain the target text after paragraph reorganization; Input the target text into a text segmenter suitable for Chinese to obtain a text segmentation result; The predetermined processing includes: S1, determine whether the current form line is a title; S2, if the current form line is a title, mark the current form line as a complete sentence; S3, if the current form line is not a title, determine whether the end of the current form line is the end symbol of a natural sentence; S4, if the end of the current form line is the end symbol of a natural sentence, mark the current form line as a complete sentence; S5, if the end of the current form line is not the end symbol of the natural sentence, then determine whether the current form line contains an intermediate connector, or the previous form line is marked as a sentence break; S6, if the current form line contains an intermediate connector, or the previous form line is marked as a sentence break, then mark the current form line as a sentence break.

2. The method according to claim 1, characterized in that The predetermined process further includes: S7: If the end of the current form line is not the end symbol of a natural sentence, the current form line does not contain an intermediate connector, and the previous form line is not marked as a sentence break, then mark the current form line as a complete sentence.

3. The method according to claim 1, characterized in that Determine whether the current form line is a title, including: Determine whether the first M characters of the current form line include a predetermined character, and the total number of characters in the current form line is less than or equal to N; wherein M and N are predetermined integers, M≤N; the predetermined character includes at least one of the following: a period, a space, an Arabic numeral, and a Chinese numeral; If the judgment result is yes, the current form behavior title is determined.

4. The method according to claim 3, characterized in that Before judging whether the current form line is a title, it also includes: Get multiple sample texts whose formal lines do not conform to the natural line typesetting rules; Extract each title from each sample text; Count the number of characters of each extracted title separately; Calculate the mean and standard deviation of the number of characters in each title; Determine a confidence interval corresponding to a predetermined confidence level; The value of N is determined according to the maximum value in the confidence interval.

5. The method according to claim 1, characterized in that The process of OCR recognition of original text or extraction of PDF text also includes: Identify the font format of each form line; the font format includes at least one of the following: font type, font size, whether it is italic, and whether it is bold; Whether each form line is a title is determined in combination with the font format of each form line.

6. The method according to claim 5, characterized in that Determine whether each form line is a title based on the font format of each form line, including: Determine whether the first M characters of the current form line include a predetermined character, and the font format of the current form line is different from the form line of the original text in a predetermined proportion; the predetermined character includes at least one of the following: a period, a space, an Arabic numeral, and a Chinese numeral; If the judgment result is yes, the current form behavior title is determined.

7. The method according to claim 1, characterized in that After inputting the target text into a text segmenter applicable to Chinese and obtaining a text segmentation result, the method further includes: Get the content of the user's question; Compare the similarity between each text segmentation result and the content of the user's question; Filter out multiple segmentation results that are most similar to the content of the user's question; Generate a prompt word according to the multiple segmentation results with the greatest similarity; The prompt word is input into the question-answering model, and the question-answering model outputs a reply to the user's question.

8. A text segmentation device, characterized in that: include: A first acquisition unit is used to acquire a target text, wherein the target text includes a plurality of format lines obtained by performing OCR recognition on an image or extracting text from a PDF file, each format line being split by a carriage return character and / or a line feed character; a marking unit, for sequentially performing predetermined processing on each form line to mark each form line as a segmented sentence or a complete sentence; The reorganization unit is used to delete the carriage return character or line feed character at the end of each line marked as a sentence break, so as to obtain the target text after paragraph reorganization; A segmentation unit, used for inputting the target text into a text segmenter suitable for Chinese to obtain a text segmentation result; The predetermined processing includes: S1, determine whether the current form line is a title; S2, if the current form line is a title, mark the current form line as a complete sentence; S3, if the current form line is not a title, determine whether the end of the current form line is the end symbol of a natural sentence; S4, if the end of the current form line is the end symbol of a natural sentence, mark the current form line as a complete sentence; S5, if the end of the current form line is not the end symbol of the natural sentence, then determine whether the current form line contains an intermediate connector, or the previous form line is marked as a sentence break; S6, if the current form line contains an intermediate connector, or the previous form line is marked as a sentence break, then mark the current form line as a sentence break.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor and the memory are communicatively connected to each other, the memory stores computer instructions, and the processor implements the text segmentation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the text segmentation method according to any one of claims 1 to 7 are implemented.

Images