Text processing device, text processing method, and recording medium

Abstract

A text processing device according to the present disclosure comprises: an insertion means for inserting, to a text composed of a plurality of sentences, a tag at a position of a delimiter of a sentence by using a language model in which a position of a delimiter of a sentence is learned by using a text for learning; a correction means for correcting, to an original character, a character of the text changed by the language model after insertion of the tag; and an output means for outputting the corrected text.

Description

Text processing device, text processing method, and recording medium

[0001] The present disclosure relates to a text processing device, a text processing method, and a recording medium.

[0002] Natural language processing is being used to analyze and aggregate natural language-based information such as documents, email or chat messages, product reviews, etc. In such cases, the target text for natural language processing may be divided into sentences.

[0003] For example, Patent Document 1 discloses extracting sentences that may have contradictory or overlapping content, and if there are multiple target documents, breaking down all of the target documents into multiple sentences.

[0004] Japanese Patent Application Laid-Open No. 2012-256107

[0005] However, when a sentence is divided by a specific character such as a period or by dependency analysis, the sentence may not be divided as expected by the user. That is, a sentence does not necessarily end with a period ".", and in chats, it may be omitted. In emails, a sentence may be written by breaking it into multiple lines to improve the format. In business systems, the number of characters that can be entered is determined by the specifications, and a sentence may be divided into multiple fields for input.

[0006] An example of an object of the present disclosure is to provide a text processing device or the like that is capable of dividing a sentence regardless of the writing method of the sentence.

[0007] A text processing device according to one aspect of the present disclosure includes an insertion means for inserting tags at the position of a sentence boundary for a text text consisting of multiple sentences using a language model that has learned the positions of sentence boundary boundaries using training text text, a correction means for correcting characters in the text text that have been changed by the language model after the tag insertion to their original characters, and an output means for outputting the corrected text text.

[0008] In one aspect of the present disclosure, a text processing method involves a computer inserting tags at the positions of sentence boundaries for a text sentence consisting of multiple sentences using a language model that has learned the positions of sentence boundaries using training text sentences, correcting characters in the text sentence that have been changed by the language model after the tag insertion to the original characters, and outputting the corrected text sentence.

[0009] In one aspect of the present disclosure, a recording medium stores a program that causes a computer to perform the following steps: insert a tag at the position of a sentence boundary for a text sentence consisting of multiple sentences using a language model that has learned the positions of sentence boundary positions using training text sentences; correct the characters in the text sentence that have been changed by the language model after the tag insertion to their original characters; and output the corrected text sentence.

[0010] According to one example of the effect of the present disclosure, it is possible to separate a sentence regardless of how the sentence is written.

[0011] FIG. 1 is a block diagram showing the configuration of a text processing device according to the present disclosure. FIG. 2 is a diagram showing a hardware configuration in which the text processing device according to the present disclosure is realized by a computer device and its peripheral devices. FIG. 3 is a diagram showing an example of a learning text sentence according to the present disclosure. FIG. 4 is a diagram showing an example of a learning text sentence according to the present disclosure. FIG. 5 is a diagram showing an example of a learning text sentence according to the present disclosure. FIG. 6 is a diagram for explaining character correction by a correction unit according to the present disclosure. FIG. 7 is an example of a sentence segment according to the present disclosure. FIG. 8 is an example of a sentence segment according to the present disclosure. FIG. 9 is an example of a sentence segment when dependency analysis is used. FIG. 10 is a flowchart showing the operation of text processing according to the present disclosure.

[0012] Hereinafter, with reference to the drawings, embodiments of a text processing device, a text processing method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail. The present embodiments do not limit the disclosed technology.

[0013] First Embodiment FIG. 1 is a block diagram showing the configuration of a text processing device 100 according to the present disclosure. As shown in FIG. 1, the text processing device 100 includes an insertion unit 101, a correction unit 102, and an output unit 103. The text processing device 100 is a device for inserting tags to separate a text sentence consisting of any number of sentences into each sentence. Examples of text sentences according to the present disclosure include, but are not limited to, sentences that do not use punctuation marks and sentences with line breaks in the middle of a sentence. Note that in this specification, "[Line Break]" refers to the position where a line break occurs in a sentence, and "[EOS (End Of Sentence)]" refers to the position where a tag is inserted. In this specification, tags are represented as [EOS], but the tag display method is not limited to this.

[0014] 2 is a diagram showing an example of a hardware configuration in which the text processing device 100 according to the present disclosure is realized by a computer device 500 including a processor. As shown in Fig. 2, the text processing device 100 includes a processor 501, memories such as a ROM (Read Only Memory) 502 and a RAM (Random Access Memory) 503, a storage device 505 such as a hard disk for storing a program 504, a communication I / F (Interface) 508 for network connection, and an input / output interface 511 for inputting and outputting data.

[0015] The processor 501 controls the entire computer device 500. The processor 501 may be, for example, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, or a combination thereof.

[0016] The processor 501 runs an operating system to control the entire text processing device 100 according to the present disclosure. The processor 501 also reads programs and data into memory from a recording medium 506 attached to a drive device 507, for example. The processor 501 also functions as the insertion unit 101, correction unit 102, and output unit 103 according to the present disclosure, or as part of these units, and executes processing or instructions in the flowchart shown in Fig. 10, which will be described later, based on the program.

[0017] The recording medium 506 is, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, or a semiconductor memory. A part of the recording medium in the storage device is a non-volatile storage device, and the program is recorded therein. The program may also be downloaded from an external computer (not shown) connected to a communication network.

[0018] The input device 509 is realized by, for example, a mouse, a keyboard, built-in key buttons, etc., and is used for input operations. The input device 509 is not limited to a mouse, a keyboard, or built-in key buttons, and may be, for example, a touch panel. The output device 510 is realized by, for example, a display, and is used to check output.

[0019] As described above, the text processing device 100 shown in Fig. 1 is realized by the computer hardware shown in Fig. 2. However, the means for realizing each unit of the text processing device 100 in Fig. 1 is not limited to the configuration described above. The text processing device 100 may be realized by a single physically coupled device, or may be realized by two or more physically separated devices connected by wire or wirelessly. For example, the input device 509 and the output device 510 may be connected to the computer device 500 via a network. The text processing device 100 shown in Fig. 1 may also be configured using cloud computing or the like.

[0020] The insertion unit 101 is a means for inserting tags into a text sentence consisting of multiple sentences at a sentence boundary position using a language model that has learned the positions of sentence boundary positions using training text sentences. The insertion unit 101 acquires the text sentence to which the tag is to be inserted, which is stored in a server device or the like, and inputs the text sentence to the language model.

[0021] The language model of the present disclosure learns the positions of sentence boundaries using training text sentences and the results of inserting tags at the positions of sentence boundaries in the training text sentences. The training text sentences may include sentences in which a single sentence spans multiple lines. The training text sentences may also be sentences used in a social networking service (SNS), email, or a business system.

[0022] 3 to 5 are diagrams showing examples of training text sentences according to the present disclosure. FIG. 3 is an example in which the training text sentence is a chat. As shown in FIG. 3, in chat, punctuation marks may be omitted, and line breaks may be inserted at intervals of a predetermined number of characters regardless of sentence boundaries. In the example of FIG. 3, characters are displayed with line breaks inserted one character at a time. FIG. 4 is an example in which the training text sentence is an email. As shown in FIG. 4, in email, a sentence may be broken into multiple lines to improve formatting. FIG. 5 is an example in which the training text sentence is a business system. In business systems, the number of characters that can be entered is determined by specifications, and a sentence may be divided into multiple entry fields. As shown in FIGS. 3 to 5, the training text sentences according to the present disclosure include line break information. Therefore, in the present disclosure, the text sentences can be divided into sentences using line break information.

[0023] As the language model, a known machine learning engine or a natural language processing algorithm can be used as appropriate. Furthermore, the language model may be a large-scale language model trained with a large amount of text data, or a transfer model obtained by transfer learning of the large-scale language model. For example, GPT-2 (Generative Pre-Training-2), GPT-3, or GPT-4 can be used as the large-scale language model. Furthermore, T5 (Text-to-Text Transfer Transformer), BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly optimized BERT approach), or ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) may be used as the large-scale language model. The language model may be stored in the storage device 505, or may be a model configured in an external system.

[0024] The correction unit 102 is a means for correcting characters of a text sentence that have been changed by the language model after the tag has been inserted back to the original characters. The correction unit 102 acquires the text sentence into which the tag output from the language model has been inserted.

[0025] When determining the position of a sentence boundary using a language model as in the present disclosure, there is a possibility that characters in the text sentence input to the language model may be changed, such as by deleting, inserting, or replacing characters. Therefore, the correction unit 102 performs an operation to restore the original characters using, for example, an edit distance or a string difference comparison tool. The correction unit 102 may, for example, use a known difference comparison tool to identify differences between the text sentence input to the language model and the text sentence output by the language model and correct the differences. Furthermore, the correction unit 102 may identify the minimum number of operations required to make the input text sentence and the text sentence output by the language model identical, based on the Levenshtein distance calculated using dynamic programming, and correct the characters in the text sentence to their original characters based on the identified operations.

[0026] FIG. 6 is a diagram for explaining character correction by the correction unit 102 in the present disclosure. Here, a method for correcting characters back to the original state based on the Levenshtein distance calculated by the correction unit 102 using dynamic programming will be described. As shown in FIG. 6 , when a text sentence such as "The way humans walk is as follows [Line Break] 1. Lift your right foot about 20 cm [Line Break]" is input to the language model, an example will be described in which "The general way humans walk is as follows [EOS] 1. Lift your right foot about 20 to 30 cm [EOS]" is output. The underlined parts in FIG. 6 are parts that have been changed from the text sentence input to the language model.

[0027] The correction unit 102 first stores, in the storage device 505 or the like, an index into which a sentence delimiter tag has been inserted from the text sentence output from the language model. An index is the number of characters from the beginning of a sentence. If the text sentence output from the language model includes multiple tags, multiple indexes are stored. For example, if the output text sentence is "The typical walking style of a human is as follows: [EOS] 1. Raise the right foot approximately 20-30 cm [EOS]," the [EOS] is inserted at the 14th and 32nd characters from the beginning of the sentence. Therefore, the indexes stored by the correction unit 102 are 14 and 32, respectively. Next, the correction unit 102 deletes the sentence delimiter tag [EOS] from the output of the language model (hereinafter also referred to as "output 2"). Next, the correction unit 102 deletes line breaks from the text sentence input to the language model (hereinafter also referred to as "input 2").

[0028] The correction unit 102 then identifies the minimum edit path and operations such as deletion, insertion, or replacement that will result in the output 2 being the input 2, for example, using dynamic programming, a well-known technique. The correction unit 102 then corrects the output 2 using the identified operation method so that the output 2 becomes the text of the input 2. Next, the correction unit 102 updates the index of each sentence delimiter tag [EOS] by adding the number of insertion operations to the index before update and subtracting the number of deletion operations from it ((updated index) = (pre-updated index) + (number of insertion operations) - (number of deletion operations)). Finally, the correction unit 102 re-inserts the sentence delimiter tag [EOS] using the updated index. The above-described method is one example of a method for correcting text sentences output from a language model to the original characters, and the correction unit 102 may also use other methods to correct the original characters.

[0029] The output unit 103 is a means for outputting the corrected text sentence. The output unit 103 displays the results of dividing the text sentence into sentences on a display device such as a display. The output unit 103 may display the results of dividing the text sentence into sentences when a user performs an operation to divide the text sentence into sentences. The output unit 103 may output the text sentence with tags inserted at the positions of the sentence dividers, or may divide the text sentence at the positions of the tags and output each sentence.

[0030] Here, an example of dividing a text sentence into sentences in the present disclosure will be described using figures. FIGS. 7 and 8 show an example of dividing a text sentence into sentences in the present disclosure. The examples in FIGS. 7 and 8 use a text sentence written in bullet points without punctuation. When the insertion unit 101 inputs the text sentence shown in (1) into a language model, the language model outputs a text sentence in which tags have been inserted and characters have been changed, as shown in (2). Next, the correction unit 102 corrects the text sentence to the original, as shown in (3). Finally, the output unit 103 outputs the text sentences divided at the tag positions, one by one, as shown in (4).

[0031] Here, using FIG. 9 , we will explain how to segment the text shown in the example of FIG. 8 into sentences using dependency analysis. Dependency analysis can be performed using natural language processing software such as OSS (Open Source Software). When using dependency analysis, preprocessing is required to concatenate multiple lines of the entire text into a single line, as shown by the underlined portion in FIG. 9 (2). Therefore, as shown in FIG. 9 (3), multiple lines are concatenated into a single line and dependency analysis is performed to determine a single sentence. In dependency analysis, the underlined portion cannot be divided because it is a valid modifier. Therefore, when determining whether a sentence is a single line, useful line break information may be lost. Therefore, when dependency analysis is used, the sentence segmentation result may differ from the expected result. On the other hand, the text processing device 100 does not require preprocessing to concatenate multiple lines of the input text into a single line. Furthermore, as described above, the text processing device 100 can use line break information as material for determining whether a sentence is a single line.

[0032] The operation of the text processing device 100 configured as above will be described with reference to the flowchart in Fig. 10. Fig. 10 is a flowchart showing an outline of the operation of the text processing device 100 in the present disclosure. The processing according to this flowchart is executed when, for example, a user performs an operation to divide an arbitrary text sentence into sentences. Note that the processing according to this flowchart may also be executed based on program control by the processor described above.

[0033] 10, the insertion unit 101 first inserts tags into a text sentence consisting of multiple sentences at the position of a sentence boundary using a language model that has learned the positions of sentence boundary using training text sentences (step S101). Next, the correction unit 102 corrects the characters in the text sentence that have been changed by the language model after the tag insertion to the original characters (step S102). Finally, the output unit 103 outputs the corrected text sentence (step S103). This completes the text processing operation of the text processing device 100.

[0034] In the text processing device 100, the insertion unit 101 inserts tags into a text sentence consisting of multiple sentences at the position of each sentence break using a language model that has learned the positions of sentence breaks using training text sentences. The correction unit 102 then corrects the characters in the text sentence that have been changed by the language model after the tag insertion back to their original characters. In this case, it is possible to break a sentence even if the text sentence to be broken does not use punctuation marks or if dependency analysis cannot break the sentence as expected. Therefore, it is possible to break a sentence regardless of the writing method of the sentence.

[0035] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0036] For example, although multiple operations are described in a sequential order in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations can be changed within the scope that does not affect the content.

[0037] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0038] (Supplementary Note 1) A text processing device comprising: an insertion means for inserting tags at the position of a sentence boundary for a text text consisting of multiple sentences using a language model that has learned the positions of sentence boundary using training text text; a correction means for correcting characters of the text text that have been changed by the language model after the insertion of the tags to their original characters; and an output means for outputting the corrected text text.

[0039] (Supplementary Note 2) The text processing device according to Supplementary Note 1, wherein the learning text sentences include sentences in which one sentence spans multiple lines.

[0040] (Supplementary Note 3) The text processing device according to Supplementary Note 2, wherein the learning text sentence is a sentence used in any of a social networking service (SNS), an email, or a business system.

[0041] (Supplementary Note 4) The text processing device according to any one of Supplementary Notes 1 to 3, wherein the language model is a large-scale language model.

[0042] (Supplementary Note 5) The text processing device according to any one of Supplementary Notes 1 to 4, wherein the correction means corrects the characters of the text sentence to the original characters using an edit distance or a character string difference comparison tool.

[0043] (Supplementary Note 6) The text processing device according to Supplementary Note 5, wherein the correction means identifies the minimum number of operations required to make the text sentence input to the language model and the text sentence output from the language model identical based on the Levenshtein distance calculated using dynamic programming, and corrects the characters of the text sentence to their original characters based on the identified operations.

[0044] (Supplementary Note 7) The text processing device according to any one of Supplementary Notes 1 to 6, wherein the output means outputs a text sentence for each sentence divided by the tag.

[0045] (Supplementary Note 8) A text processing method in which a computer inserts tags at the positions of sentence breaks for a text sentence consisting of multiple sentences using a language model that has been trained to learn the positions of sentence breaks using training text sentences, corrects characters in the text sentence that have been changed by the language model after the tag insertion to their original characters, and outputs the corrected text sentence.

[0046] (Supplementary Note 9) The text processing method according to Supplementary Note 8, wherein the learning text includes a sentence in which one sentence spans multiple lines.

[0047] (Supplementary Note 10) The text processing method according to Supplementary Note 9, wherein the learning text is a text used in any of a social networking service (SNS), email, or business system.

[0048] (Supplementary Note 11) The text processing method according to any one of Supplementary Notes 8 to 10, wherein the language model is a large-scale language model.

[0049] (Supplementary Note 12) The text processing method according to any one of Supplementary Notes 8 to 11, wherein characters of the text sentence are corrected to original characters using an edit distance or character string difference comparison tool.

[0050] (Supplementary Note 13) A text processing method according to Supplementary Note 12, further comprising: identifying a minimum number of operations required to make a text sentence input to the language model identical to a text sentence output from the language model based on the Levenshtein distance calculated using dynamic programming; and correcting characters in the text sentence to their original characters based on the identified operations.

[0051] (Supplementary Note 14) The text processing method according to any one of Supplementary Notes 8 to 13, wherein a text document is output for each sentence divided by the tag.

[0052] (Supplementary Note 15) A recording medium storing a program that causes a computer to execute the following steps: inserting tags at the positions of sentence boundaries for a text sentence consisting of multiple sentences using a language model that has learned the positions of sentence boundaries using training text sentences; correcting characters in the text sentence that have been changed by the language model after the tag insertion to their original characters; and outputting the corrected text sentence.

[0053] (Supplementary Note 16) The recording medium according to Supplementary Note 15, wherein the learning text sentences include sentences in which one sentence spans multiple lines.

[0054] (Supplementary Note 17) The recording medium according to Supplementary Note 16, wherein the learning text sentence is a sentence used in any of a social networking service (SNS), email, or business system.

[0055] (Supplementary Note 18) The recording medium according to any one of Supplementary Notes 15 to 17, wherein the language model is a large-scale language model.

[0056] (Supplementary Note 19) The recording medium according to any one of Supplementary Notes 15 to 18, wherein the characters of the text sentence are corrected to their original characters using an edit distance or character string difference comparison tool.

[0057] (Supplementary Note 20) The recording medium according to any one of Supplementary Notes 15 to 19, wherein a text document is output for each sentence divided by the tag.

[0058] Some or all of the configurations described in Supplements 2 to 7 that are dependent on Supplement 1 above may also be dependent on Supplements 8 and 15 in the same dependency relationship as Supplements 2 to 7. Not limited to Supplements 1, 8, and 15, some or all of the configurations described as Supplements may be made dependent on various hardware, software, various recording devices for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0059] 100 Text processing device 101 Insertion unit 102 Correction unit 103 Output unit 500 Computer device 501 Processor 502 ROM 503 RAM 504 Program 505 Storage device 506 Recording medium 507 Drive device 508 Communication interface 511 Input / output interface 512 Bus

Claims

1. A text processing device comprising: an insertion means for inserting tags at the position of a sentence boundary for a text text consisting of multiple sentences using a language model that has learned the positions of sentence boundary boundaries using training text text; a correction means for correcting characters in the text text that have been changed by the language model after the insertion of the tags to their original characters; and an output means for outputting the corrected text text.

2. The text processing device according to claim 1, wherein the learning text includes a sentence that spans multiple lines.

3. The text processing device according to claim 2, wherein the learning text is a text used in any of a social networking service (SNS), email, or business system.

4. The text processing device according to any one of claims 1 to 3, wherein the language model is a large-scale language model.

5. A text processing device according to any one of claims 1 to 4, wherein the correction means corrects the characters of the text sentence to the original characters using an edit distance or a character string difference comparison tool.

6. The text processing device according to claim 5, wherein the correction means identifies the minimum number of operations required to make the text sentence input to the language model and the text sentence output from the language model identical based on the Levenshtein distance calculated using dynamic programming, and corrects the characters of the text sentence to their original characters based on the identified operations.

7. The text processing device according to any one of claims 1 to 6, wherein said output means outputs a text sentence for each sentence divided by said tag.

8. A text processing method in which a computer inserts tags at the positions of sentence boundaries for a text sentence consisting of multiple sentences using a language model that has been trained to determine the positions of sentence boundaries using training text sentences, corrects characters in the text sentence that have been changed by the language model after the tag insertion to their original characters, and outputs the corrected text sentence.

9. The text processing method according to claim 8, wherein the learning text includes a sentence that spans multiple lines.

10. The text processing method according to claim 9, wherein the learning text is a text used in any of a social networking service (SNS), email, or business system.

11. A text processing method according to any one of claims 8 to 10, wherein the language model is a large-scale language model.

12. The text processing method according to any one of claims 8 to 11, wherein the characters of the text sentence are corrected to the original characters using an edit distance or a string difference comparison tool.

13. The text processing method according to claim 12, further comprising: identifying the minimum number of operations required to make a text sentence input to the language model identical to a text sentence output from the language model based on the Levenshtein distance calculated using dynamic programming; and correcting characters in the text sentence to their original characters based on the identified operations.

14. A text processing method according to any one of claims 8 to 13, wherein a text sentence is output for each sentence divided by the tag.

15. A recording medium storing a program that causes a computer to execute the following steps: for a text sentence consisting of multiple sentences, insert a tag at the position of a sentence boundary using a language model that has learned the positions of sentence boundary using training text sentences; correct the characters of the text sentence that have been changed by the language model after the tag insertion to the original characters; and output the corrected text sentence.

16. The recording medium according to claim 15, wherein the learning text sentences include sentences in which one sentence spans multiple lines.

17. The recording medium according to claim 16, wherein the learning text sentence is a sentence used in any of SNS (Social Networking Service), email, and business systems.

18. The recording medium according to any one of claims 15 to 17, wherein the language model is a large-scale language model.

19. The recording medium according to any one of claims 15 to 18, wherein the characters of the text sentence are corrected to the original characters using an edit distance or character string difference comparison tool.

20. A recording medium according to any one of claims 15 to 19, which outputs a text document for each sentence divided by the tag.

Images