Natural language processing system, natural language processing method, and natural language processing program

Abstract

To provide a natural language processing system, method, and program which exhibit suppressed data capacity and achieve high processing speed.SOLUTION: A natural language processing system includes: a second weight arrangement calculation unit which, on the basis of a first weight arrangement produced by learning with learning character strings having presence or absence of a word boundary set in at least one character space and two or more origins corresponding to any character spaces of the learning character strings, integrates weights of at least one set of the two or more origins corresponding to characters or character strings having an identical or inclusive relation to calculate a second weight arrangement; an origin extraction unit which, on the basis of window width information on an extraction range set in advance, extracts at least one origin corresponding to a character space of a target character string including two or more characters; and a boundary determination unit which, on the basis of a learned model produced by the learning, the second weight arrangement, and the at least one extracted origin, determines presence or absence of a word boundary in the character space corresponding to the at least one extracted origin in the target character string.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a natural language processing system, a natural language processing method, and a natural language processing program. [Background technology]

[0002] In recent years, computer-based natural language processing technologies such as contextual analysis and semantic analysis of text have been developing.

[0003] In computer-based natural language processing, determining word boundaries is essential when dividing a sentence into words. In other words, natural language processing is difficult for languages ​​and sentences in which determining word boundaries is difficult.

[0004] Determining word boundaries in languages ​​that do not explicitly mark word boundaries, such as Japanese, Chinese, and Korean, is more difficult than in languages ​​that explicitly mark word boundaries, such as English and German. Even in languages ​​that explicitly mark word boundaries, determining word boundaries in sentences that include compound nouns or compound words that combine two or more words is difficult.

[0005] Patent Document 1 describes an example of a method for accurately determining word boundaries even in languages ​​or sentences where determining word boundaries is difficult.

[0006] The word segmentation device described in Patent Document 1 extracts features for each estimated word segmentation position in each string included in a set of strings in a target field, and features for each space between characters in each string included in a set of word-segmented strings in a field different from the target field, and determines the word segmentation positions in the strings in the target field based on a word segmentation model trained based on the extracted features. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6269953 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the word segmentation device described in Patent Document 1 has problems such as the excessively large data capacity of the device, which means that processing is mainly done on the server, raising security concerns, and the fact that word segmentation processing takes a long time.

[0009] Therefore, an object of the present invention is to provide a natural language processing system, a natural language processing method, and a natural language processing program that can reduce data volume and achieve high processing speed. [Means for solving the problem]

[0010] A natural language processing system according to one embodiment of the present invention includes: a second weight array calculation unit that calculates a second weight array by integrating weights of at least one set of two or more features corresponding to identical or inclusive characters or character strings based on a first weight array regarding weights for determining word boundaries of a string for all or part of the two or more features generated by learning using at least one training string in which the presence or absence of a word boundary between at least one character is set and two or more features corresponding to any character intervals in the training string; a feature extraction unit that extracts at least one feature corresponding to a character interval between characters of a target string containing two or more characters based on window width information regarding an extraction range that is set in advance; and a boundary determination unit that determines the presence or absence of a word boundary between characters in the target string that corresponds to the at least one extracted feature based on the trained model generated by learning, the second weight array, and the at least one extracted feature.

[0011] A natural language processing method according to one embodiment of the present invention involves a computer generating a training string using at least one training string in which the presence or absence of a word boundary between at least one character is set, and two or more features corresponding to any character spaces in the training string. The computer generates a second weight array based on a first weight array relating to weights for determining word boundaries of the string for all or part of the two or more features. The second weight array is then used to calculate weights for at least one set of two or more features corresponding to identical or inclusive characters or strings. The computer then extracts at least one feature corresponding to the character spaces in a target string containing two or more characters based on window width information relating to a predetermined extraction range. The computer then determines whether or not a word boundary exists between the characters in the target string that corresponds to the extracted at least one feature, based on the trained model generated by training, the second weight array, and the at least one extracted feature.

[0012] A natural language processing program according to one embodiment of the present invention includes a computer that implements the following: a second weight array calculation unit that calculates a second weight array by integrating weights of at least one set of two or more features corresponding to identical or inclusive characters or character strings based on a first weight array related to weights for determining word boundaries of a string for all or part of the two or more features generated by training using at least one training string in which the presence or absence of a word boundary between at least one character is set and two or more features corresponding to any character intervals in the training string; a feature extraction unit that extracts at least one feature corresponding to a character interval in a target string containing two or more characters based on window width information related to an extraction range that is set in advance; and a boundary determination unit that determines the presence or absence of a word boundary between characters in the target string that corresponds to the at least one extracted feature based on the trained model generated by training, the second weight array, and the at least one extracted feature.

[0013] In this invention, a "unit" does not simply mean a physical means, but also includes cases where the functions of the "unit" are realized by software. Furthermore, the functions of one "unit" or device may be realized by two or more physical means or devices, and the functions of two or more "units" or devices may be realized by one physical means or device. [Effects of the Invention]

[0014] According to the present invention, it is possible to provide a natural language processing system, a natural language processing method, and a natural language processing program that can reduce data volume and achieve high processing speed. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram showing a configuration of a natural language processing system 100 according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an overview of processing in a natural language processing system 100. [Figure 3] 10 is a diagram showing an example of information stored in a learning dictionary storage unit 122. FIG. [Figure 4] 10 is a diagram showing an example of information stored in a learning character string storage unit 124. FIG. [Figure 5] FIG. 10 is a diagram showing examples of features extracted by the feature extraction unit 125. [Figure 6A] 10 is a diagram showing an example of information stored in a first weight array storage unit 127. FIG. [Figure 6B] 10 is a diagram showing an example of information stored in a first weight array storage unit 127. FIG. [Figure 6C] 10 is a diagram showing an example of information stored in a first weight array storage unit 127. FIG. [Figure 7A] FIG. 1 is a diagram illustrating an outline of a process in which a feature corresponding to a space between characters is extracted by the natural language processing system 100. [Figure 7B] FIG. 1 is a diagram illustrating an outline of a process in which a feature corresponding to a space between characters is extracted by the natural language processing system 100. [Figure 8A]10 is a diagram showing an example of information stored in a second weight array storage unit 132. FIG. [Figure 8B] 10 is a diagram showing an example of information stored in a second weight array storage unit 132. FIG. [Figure 9] 10 is a diagram showing an example of information stored in a target character string storage unit 134. FIG. [Figure 10] 10 is a diagram showing an example of information stored in a divided character string storage unit 136. FIG. [Figure 11] 10 is a diagram showing an example of information stored in a determined character string storage unit 138. FIG. [Figure 12] 10 is a diagram showing experimental results regarding the processing speed in the natural language processing system 100. FIG. [Figure 13] FIG. 13 is a diagram showing an example of a contract 1300 to be processed by the natural language processing system 100. [Figure 14] 10 is a flowchart showing an example of a calculation process of a second weight array in the natural language processing system 100. [Figure 15] 10 is a flowchart showing an example of a process for determining a word boundary in the natural language processing system 100. DETAILED DESCRIPTION OF THE INVENTION

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described with reference to the accompanying drawings, in which: Figure 1 is a diagram showing the configuration of a natural language processing system 100 according to an embodiment of the present invention;

[0017] The natural language processing system 100 is a system that is communicably connected to an information processing system 111 and a user terminal 112 via a network such as the Internet. Details of the natural language processing system 100 will be described later.

[0018] The information processing system 111 is an information processing system that stores a training dictionary and training strings and provides the training dictionaries and training strings to an external information processing system. The information processing system 111 can provide the natural language processing system 100 with training dictionary information related to the training dictionary, training string information related to the training strings, and information related to features corresponding to spaces between characters in the training strings.

[0019] The user terminal 112 is a computer used by a user, such as a smartphone, a tablet terminal, a personal computer, etc. The user terminal 112 can provide the natural language processing system 100 with document data or image data that includes character strings or data related to character strings that are the subject of word boundary determination by the natural language processing system 100, or can instruct the natural language processing system 100 to obtain such data from an external information processing system.

[0020] Although FIG. 1 shows three user terminals 112a, 112b, and 112c as an example of the user terminal 112, the number of user terminals 112 is not limited to this.

[0021] Furthermore, the natural language processing system 100 may be a system that is partly or entirely incorporated into the information processing system 111, or may be a system that is incorporated into the user terminal 112.

[0022] FIG. 2 is a diagram showing an outline of the processing in the natural language processing system 100. As shown in FIG.

[0023] The natural language processing system 100 is a system that determines whether or not there is a word boundary between characters in an arbitrary character string (target character string). In other words, the natural language processing system 100 determines whether or not the characters to be determined are a word boundary (the start position or end position of a word).

[0024] First, the natural language processing system 100 learns using a training string in which the presence or absence of word boundaries between characters is set, features corresponding to any spaces between characters in the training string, and a training dictionary, and generates a trained model and a first weight array (S201).

[0025] Here, the feature is information corresponding to the space between characters in a string, and includes, for example, character n-grams, character type n-grams, and dictionary word features. The first weight array is array data that associates features with weights that serve as criteria for determining whether or not there is a word boundary.

[0026] Next, the natural language processing system 100 calculates a second weight array by integrating the weights of features that correspond to the same or inclusive characters or character strings included in the first weight array based on the first weight array (S202). For example, the natural language processing system 100 integrates the weights of the two features "the whole world" and "the world" included in the first weight array so that they are included in the weight of the single feature "the whole world," thereby generating the second weight array.

[0027] Thereafter, the natural language processing system 100 extracts features corresponding to the characters between which word boundaries are to be determined in the target character string (S203).

[0028] Then, the natural language processing system 100 determines whether or not there is a word boundary between the characters based on the trained model, the second weight array, and the extracted features corresponding to the characters between which word boundaries are to be determined in the target string (S204).

[0029] The natural language processing system 100 can determine whether or not there is a word boundary for any character string. That is, the natural language processing system 100 can also determine word boundaries for character strings in specialized fields, and for example, can also determine word boundaries for character strings included in legal documents such as contracts.

[0030] Next, the natural language processing system 100 will be described in detail. As shown in Fig. 1, the natural language processing system 100 includes a training dictionary acquisition unit 121, a training dictionary storage unit 122, a training string acquisition unit 123, a training string storage unit 124, a feature extraction unit 125, a learning unit 126, a first weight array storage unit 127, a trained model storage unit 128, a second weight array calculation unit 131, a second weight array storage unit 132, a target string acquisition unit 133, a target string storage unit 134, a string division unit 135, a divided string storage unit 136, a boundary determination unit 137, a determined string storage unit 138, an output unit 141, a sentence division unit 142, a sentence type determination unit 143, and a display unit 144. The computer constituting the natural language processing system 100 includes a processor and a storage area. Each unit shown in Fig. 1 can be realized, for example, by using the storage area or by the processor executing a program stored in the storage area.

[0031] The learning dictionary acquisition unit 121 acquires learning dictionary information relating to a learning dictionary containing at least one word from the information processing system 111 and stores it in the learning dictionary storage unit 122 .

[0032] Here, the learning dictionary information is information related to a dictionary that includes common words, and is information that the natural language processing system 100 learns when generating a first weight array, which will be described later.

[0033] In addition, the learning dictionary acquisition unit 121 may acquire the learning dictionary information itself and store it in the learning dictionary storage unit 122, or may acquire information regarding the links to the learning dictionary information and store the information regarding the links to the learning dictionary information in the learning dictionary storage unit 122.

[0034] 3 is a diagram showing an example of information stored in the learning dictionary storage unit 122. The information stored in the learning dictionary storage unit 122 includes, for example, learning dictionary information, and the learning dictionary information includes, for example, learning word IDs and learning word information. The learning word IDs are learning word identification information that identify words (learning words) included in the learning dictionary. The learning word information is information related to learning words.

[0035] The learning dictionary storage unit 122 may store the learning dictionary information itself, or may store information relating to links to the learning dictionary information.

[0036] The training string acquisition unit 123 acquires training string information regarding a training string, which is at least one string in which the presence or absence of a word boundary is set between at least one character, from the information processing system 111, and stores the information in the training string memory unit 124.

[0037] Here, the training string information is information about character strings for which the presence or absence of word boundaries has already been set, and is so-called training data in machine learning. The training string information is also information that the natural language processing system 100 learns when generating a first weight array, which will be described later.

[0038] Note that character spacing includes the space between characters (for example, between "zen" and "se" in the string "zenkai" (the whole world)) and the start and end positions of the string (for example, the left side of "zen" and the right side of "kai" (the whole world) in the string "zenkai" (the whole world)).

[0039] Furthermore, the training string information only needs to set whether there is a word boundary between at least one character, and does not need to set whether there is a word boundary between all characters. Furthermore, the training string information may use, for example, " / " as information indicating the position of a word boundary in a training string, or other symbols or methods may be used.

[0040] In addition, the learning string acquisition unit 123 may acquire the learning string information itself and store it in the learning string storage unit 124, or may acquire information related to the link of the learning string information and store the information related to the link of the learning string information in the learning string storage unit 124.

[0041] 4 is a diagram showing an example of information stored in the learning string storage unit 124. The information stored in the learning string storage unit 124 includes, for example, a learning string ID and learning string information. The learning string ID is learning string identification information that identifies a learning string included in the learning string information. The learning string information is information related to a learning string.

[0042] In Figure 4, the positions of word boundaries are indicated using " / " between characters that form word boundaries in the training string information, but the information indicating the positions of word boundaries is not limited to " / " and other symbols or methods may be used.

[0043] The feature extraction unit 125 extracts features corresponding to spaces between characters in a string containing two or more characters based on window width information about a window width that indicates a preset extraction range, and generates feature information. Here, the window width information is information that indicates the range extracted by the feature extraction unit 125. Details of features will be described later.

[0044] The feature extraction unit 125 can extract two or more features corresponding to a space between characters in the training string information stored in the training string storage unit 124. The feature extraction unit 125 can also extract at least one feature corresponding to a space between characters in the target string information stored in the target string storage unit 134, and at least one feature corresponding to a space between characters in the divided string information stored in the divided string storage unit 136, which will be described later.

[0045] When the feature extraction unit 125 extracts two or more features corresponding to character spaces in the training string information, the feature extraction unit 125 may extract features corresponding to character spaces for which the presence or absence of a word boundary is set, or may extract features corresponding to character spaces for which the presence or absence of a word boundary is not set. Furthermore, the feature extraction unit 125 may extract two or more features from one character space, or may extract one feature each from two or more character spaces (two or more features in total).

[0046] The feature extraction unit 125 may obtain feature information on two or more features corresponding to the spaces between characters in the learning string information from the information processing system 111, and generate feature information.

[0047] Next, a detailed description will be given of the features extracted by the feature extraction unit 125. The features extracted by the feature extraction unit 125 may include, for example, character n-grams, character type n-grams, and dictionary word features.

[0048] A character n-gram is information that indicates a character or a string between a predetermined character space in a string and a predetermined relative position from the predetermined character space, where "n" indicates the number of characters in the character or string.

[0049] A character type n-gram is information indicating the character type of a character or a character string between a predetermined character space in a character string and a predetermined relative position starting from the predetermined character space. Here, character type is information indicating the type of character, and includes, for example, kanji, katakana, hiragana, Roman letters, numbers, and others. Here, "n" indicates the number of characters in the character or string.

[0050] A dictionary word feature is information that indicates a word that is identical to the training word information included in the training dictionary information, among the characters or character strings that are located between a specified character space in a character string and a specified relative position starting from the specified character space.

[0051] Fig. 5 is a diagram showing an example of a feature extracted by the feature extraction unit 125. Fig. 5 shows an example of a feature corresponding to the character space 501 between the characters "kai" and "no" in the character string "Warera wa sekai no kokumin ga, (We are all citizens of the whole world)."

[0052] First, the feature extraction unit 125 sets a window width based on window width information relating to the window width that indicates the range for feature extraction. Fig. 5 shows window width 502, which is set to 3 (i.e., three characters before and after character spacing 501). The window width information only needs to be set before the feature extraction unit 125 performs extraction processing.

[0053] Then, feature extraction unit 125 extracts character n-grams and character type n-grams for each relative position from character space 501 (in the case of FIG. 5 where the window width is 3, a total of six relative positions from relative positions "-3" to "2"). Note that because the window width is 3, the character n-grams and character type n-grams corresponding to each relative position are character 1-gram to character 3-gram and character type 1-gram to character type 3-gram, respectively.

[0054] As shown in the example in FIG. 5, the character 1-grams include, in order of relative position, "zen," "yo," "kai," "no," "koku," and "min."

[0055] Furthermore, the character type 1-gram includes, in order of relative position, "Kan", "Kan", "Kan", "Hi", "Kan", and "Kan". Here, the character type "Kan" indicates that the character type is Kanji, and the character type "Hi" indicates that the character type is Hiragana. Note that the method of notating character types is not limited to the method shown in FIG. 5.

[0056] Furthermore, if the learning dictionary information contains only three learning word information items, namely, "human beings," "citizens," and "the whole world," the dictionary word features will be only "citizens" (n=2) and "the whole world" (n=3).

[0057] The learning unit 126 learns the relationship between features and the presence or absence of word boundaries using the training string information stored in the training string storage unit 124 and two or more feature information items corresponding to the spaces between characters in the training string extracted and generated by the feature extraction unit 125, generates a first weight array and a trained model, stores the first weight array information related to the generated first weight array in the first weight array storage unit 127, and stores the generated trained model in the trained model storage unit 128.

[0058] In addition, the learning unit 126 may learn the relationship between features and the presence or absence of word boundaries using the learning dictionary information stored in the learning dictionary storage unit 122, the learning string information stored in the learning string storage unit 124, and two or more feature information items corresponding to the spaces between characters in the learning string, extracted and generated by the feature extraction unit 125.

[0059] Here, the first weight array is array data relating to weights that serve as a criterion for determining whether or not a character boundary exists in a string, and is array data that corresponds two or more features corresponding to character spaces in the training string with the weights for each relative position of each of the two or more features corresponding to character spaces in the training string.

[0060] The two or more features corresponding to the character spaces of the training strings included in the first weight array may be all or part of the features corresponding to the feature information generated by the feature extraction unit 125. Furthermore, the two or more features corresponding to the character spaces of the training strings included in the first weight array may be all or part of the feature information acquired by the feature extraction unit 125 from the information processing system 111.

[0061] 6A to 6C are diagrams showing examples of information stored in first weight array storage unit 127. The information stored in first weight array storage unit 127 includes, for example, first weight array information, which includes, for example, feature information, relative position information, and weight information (character n-gram weight information indicating weights related to character n-grams, character type n-gram weight information indicating weights related to character type n-grams, and dictionary word feature weight information indicating weights related to dictionary word features). As shown in FIGS. 6A to 6C, first weight array storage unit 127 may store first weight array information for each type of weight information, or may store multiple types of weight information in association with each type of feature information.

[0062] Here, feature information is information about two or more features corresponding to character spaces in a training string, extracted and generated by the feature extraction unit 125. Relative position information is information about the relative position of each feature corresponding to feature information relative to the character spaces. Weight information is information about weights that correspond to features and relative positions and serve as a criterion for determining whether or not a string has a word boundary.

[0063] The feature information stored in the first weight array storage unit 127 may be information about all or some of the features extracted by the feature extraction unit 125. Furthermore, the feature information stored in the first weight array storage unit 127 may be all or some of the feature information acquired by the feature extraction unit 125 from the information processing system 111.

[0064] The generation of the first weight array by the learning unit 126 is executed, for example, in response to an instruction from a system administrator. Furthermore, the learning may be executed automatically at a predetermined timing such as during system maintenance, or may be executed immediately before the calculation process of the second weight array by the second weight array calculation unit 131, which will be described later. The learning algorithm is not particularly limited, and may be, for example, a random forest or an SVM (support vector machine).

[0065] In addition, the first weight array storage unit 127 may store the first weight array generated by the learning unit 126, or may store information corresponding to the first weight array stored in an external information processing system or information regarding links to the information corresponding to the first weight array.

[0066] The trained model storage unit 128 stores trained models trained by the training unit 126. The trained models trained by the training unit 126 are models that determine whether or not there is a word boundary between any characters, based on features and weight arrays (a first weight array and a second weight array, which will be described later) corresponding to the characters.

[0067] Specifically, the trained model outputs a score regarding whether or not there is a word boundary between the characters to be determined based on the input features and weight arrays (the first weight array and the second weight array described below) corresponding to specific character intervals.

[0068] The trained model storage unit 128 may store trained models generated by the training unit 126, or may store information corresponding to trained models stored in an external information processing system or information regarding links to information corresponding to the trained models.

[0069] The second weight array calculation unit 131 calculates a second weight array by integrating the weights of at least one set of two or more features corresponding to identical or inclusive characters or character strings based on the first weight array information stored in the first weight array storage unit 127, and stores the second weight array information regarding the generated second weight array in the second weight array storage unit 132.

[0070] Specifically, for example, the character n-gram "sekai" and the dictionary word feature "sekai" included in the first weight array are features corresponding to the same character string "sekai." The second weight array calculation unit 131 can combine the weights of the character n-gram "sekai" and the dictionary word feature "sekai" corresponding to the same character string to calculate the weight of the feature "sekai" and generate the second weight array. Note that when combining weights of features corresponding to the same character or character string, the second weight array calculation unit 131 combines weight information of different types.

[0071] For example, if the features included in the first weight array include the feature "the whole world", the feature "all", and the feature "the world", the features "the whole world" and "all", and the features "the whole world" and "the world" are in an inclusive relationship. The second weight array calculation unit 131 can integrate the weights of the three inclusive features to calculate the weight of the feature "the whole world" and generate the second weight array.

[0072] When integrating weights of features in an inclusive relationship, the second weight array calculation unit 131 integrates the weights for each type of weight information. That is, when the weight information stored in the first weight array storage unit 127 includes both character n-gram weight information and character type n-gram weight information, the second weight array calculation unit 131 integrates the character n-gram weight information of at least one set of two or more features in an inclusive relationship, and also integrates the character type n-gram weight information of at least one set of two or more features in an inclusive relationship.

[0073] The process by the second weight array calculation unit 131 to combine the weights of at least one set of two or more features corresponding to the same or inclusive characters or character strings may be a predetermined calculation process performed in the process of determining whether or not there is a word boundary by the boundary determination unit 137, which will be described later. That is, for example, the second weight array calculation unit 131 may combine the weights of at least one set of two or more features corresponding to the same or inclusive characters or character strings included in the first weight array by adding the weights of the at least one set of two or more features, or may combine the weights of at least one set of two or more features corresponding to the same or inclusive characters or character strings included in the first weight array by averaging the weights of the at least one set of two or more features. Other calculation processes may also be used.

[0074] It should be noted that the second weight array calculation unit 131 does not need to integrate all combinations of features corresponding to identical or inclusive characters or character strings. The calculation of the second weight array by the second weight array calculation unit 131 is executed, for example, in response to an instruction from a system administrator. The calculation process may be executed automatically at a predetermined timing, such as during system maintenance, or may be executed immediately before the determination process by the boundary determination unit 137, which will be described later.

[0075] The weight integration process by the second weight array calculation unit 131 will be described with reference to Figures 7A and 7B. Figures 7A and 7B are diagrams showing an overview of the process in which features corresponding to spaces between characters are extracted by the natural language processing system 100. Figure 7A is a diagram showing an overview of the process in the case where weights are not integrated by the second weight array calculation unit 131, and Figure 7B is a diagram showing an overview of the process in the case where weights are integrated by the second weight array calculation unit 131.

[0076] Node 701 indicates a character space in a character or character string to be processed by the natural language processing system 100, edge 702 indicates that nodes 701 are connected by a character, and character 703 indicates the character corresponding to edge 702. Feature 704 indicates a feature corresponding to node 701 (i.e., a character space). For example, node 701a indicates the rightmost character space (i.e., the right end of the character) in the character string "sekai" (world), and indicates that the features corresponding to node 701a are feature 704a (feature "zenkai") and feature 704b (feature "sekai"). The boundary determination unit 137, which will be described later, determines whether or not there is a word boundary based on weight information corresponding to feature 704a and feature 704b, respectively. Note that the extracted features are not limited to feature 704 shown in FIGS. 7A and 7B .

[0077] 7A, when weights are not integrated by the second weight array calculation unit 131, a feature 704a (feature "the whole world") and a feature 704b (feature "the world") correspond to node 701a. In other words, when weights are not integrated by the second weight array calculation unit 131, the presence or absence of a word boundary is determined based on the weight information corresponding to each of feature 704a (feature "the whole world") and feature 704b (feature "the world").

[0078] 7B, the second weight array calculation unit 131 can combine the weights of feature 704a (feature "the whole world") and feature 704b (feature "the world"), which correspond to strings in an inclusive relationship. In other words, as shown in FIG. 7B, when the weights are combined by the second weight array calculation unit 131, the presence or absence of a word boundary is determined based on the weight information corresponding to the combined feature 704c.

[0079] As a result, the boundary determination unit 137, which will be described later, can refer to the weight information corresponding to the integrated feature "the whole world" when determining whether or not there is a word boundary in the string "the whole world," for example, and can do so without referring to the weight information corresponding to each of the features "the world."

[0080] The second weight array information includes weight information of the features used for integration (e.g., the feature "all" and the feature "world"), and the boundary determination unit 137 can refer to the weights of the features used for integration (e.g., the feature "all" and the feature "world") included in the second weight array information. That is, the boundary determination unit 137 can refer to the weights of the feature "all" and the feature "world" when determining whether or not there is a word boundary in, for example, a character string "all citizens of the world" (i.e., a character string that is not the character string "the whole world" but includes the character string "all" and the character string "world").

[0081] 8A and 8B are diagrams showing examples of information stored in the second weight array storage unit 132. The information stored in the second weight array storage unit 132 includes, for example, second weight array information, which includes, for example, feature information, relative position information, and weight information (character n-gram weight information, character type n-gram weight information). As shown in FIGS. 8A and 8B, the second weight array storage unit 132 may store second weight array information for each piece of weight information, or may store weight information for each feature in association with each piece of feature information.

[0082] The target string acquisition unit 133 acquires target string information, which is a string that is the target of word boundary determination, based on a string provided to the natural language processing system 100 or document data or image data that includes data related to the string, and stores the information in the target string storage unit 134.

[0083] The target character string acquisition unit 133 can acquire character strings contained in a document, for example, a legal document such as a contract, from the document data or image data of the document.

[0084] The target string acquisition unit 133 may receive document data or image data containing a string or data related to a string from the user terminal 112, or may acquire document data or image data from an external information processing system based on instructions from a user via the user terminal 112.

[0085] When the natural language processing system 100 receives document data that includes data related to character strings, the target character string acquisition unit 133, for example, analyzes the document data and acquires the character strings included in the received document data.

[0086] When the natural language processing system 100 receives image data including data related to a character string, the target character string acquisition unit 133 acquires the character string included in the received image data, for example, using image recognition technology.

[0087] 9 is a diagram showing an example of information stored in the target string storage unit 134. The information stored in the target string storage unit 134 includes, for example, a target string ID and target string information. The target string ID is target string identification information that identifies a target string that is the target of word boundary determination in the natural language processing system 100. The target string information is information related to a target string that is the target of word boundary determination in the natural language processing system 100.

[0088] The character string dividing unit 135 divides the target character string stored in the target character string storage unit 134 to generate divided character strings, and stores divided character string information relating to the generated divided character strings in the divided character string storage unit 136 .

[0089] The character string dividing unit 135 can divide the target character string at any inter-character position depending on the computing environment of the computer and the properties of the target character string. The character string dividing unit 135 may divide the target character string based on the inter-character position to be divided that is set according to predetermined conditions, or may divide the target character string based on the inter-character position to be divided that is set by a user operation via the user terminal 112.

[0090] The character string dividing unit 135 may divide the target character string so that the divided character strings do not overlap each other, or may divide the target character string so that the divided character strings partially overlap each other.

[0091] In addition, in the parallel determination process described below in which the presence or absence of word boundaries is determined in parallel for each divided string, in order to avoid cross-referencing between the divided strings, the string dividing unit 135 may divide the target string into divided strings so that both ends of the divided strings overlap by an amount corresponding to the window width information.

[0092] 10 is a diagram showing an example of information stored in divided character string storage unit 136. The information stored in divided character string storage unit 136 includes, for example, divided character string IDs and divided character string information. The divided character string IDs are divided character string identification information that identifies divided character strings divided by character string dividing unit 135. The divided character string information is information related to divided character strings.

[0093] In Figure 10, the split strings are shown so that both ends of the split strings overlap by the window width corresponding to the window width information in order to avoid references between the split strings in the parallel determination process described below, which determines whether or not there is a word boundary for each split string in parallel.

[0094] The boundary determination unit 137 determines whether or not there is a word boundary between characters in the target string or the divided string that corresponds to at least one extracted feature, based on the trained model stored in the trained model storage unit 128, the second weight array information stored in the second weight array storage unit 132, and at least one feature information corresponding to between characters in the target string or the divided string that is extracted and generated by the feature extraction unit 125, and stores determined string information in which a word boundary is set between at least one character of the target string in the determined string storage unit 138.

[0095] Specifically, the boundary determination unit 137 inputs the second weight array information and at least one feature information corresponding to a space between characters in the target string or the divided string to the trained model, and calculates a score regarding the presence or absence of a word boundary between the characters to be determined. Then, if the score is equal to or greater than a certain value, the boundary determination unit 137 determines that the space between the characters is a word boundary.

[0096] Furthermore, the boundary determination unit 137 can determine the presence or absence of word boundaries between characters corresponding to at least one extracted feature based on a score calculated based on the window width information, regarding the presence or absence of word boundaries in all or some combinations of characters or all or some combinations of character types for the extraction range indicated by the window width information. In other words, the boundary determination unit 137 or another information processing system can calculate in advance scores for character strings or character type strings that may be included in the extraction range (pre-score calculation process), and the boundary determination unit 137 can use the pre-calculated scores to determine the presence or absence of word boundaries between characters corresponding to at least one extracted feature.

[0097] As described above, boundary determination unit 137 determines whether or not there is a word boundary between characters based on features corresponding to characters or character strings in the extraction range indicated by the window width information (for example, six characters when the window width is three). Here, for example, if the feature is a character type n-gram, character type n-grams are made up of a combination of a finite number of types (for example, kanji, katakana, hiragana, roman letters, numbers, and other, a total of six types), and therefore the combinations of character type n-grams corresponding to characters or character strings in the extraction range indicated by the window width information are finite.

[0098] Therefore, the boundary determination unit 137 or another information processing system calculates in advance a score for the presence or absence of a word boundary for each combination of character types within the extraction range indicated by the window width information. If the character types of a character string included in the extraction range corresponding to the character gap to be determined are a combination of character types calculated in advance, the boundary determination unit 137 uses the score calculated in advance to determine the presence or absence of a word boundary between the characters. In this case, the boundary determination unit 137 can determine the presence or absence of a word boundary between the characters using the score calculated in advance based on the character type n-gram and a score calculated based on other features. This allows the boundary determination unit 137 to more quickly determine the presence or absence of a word boundary between the characters.

[0099] The above pre-score calculation process may be performed when the feature is not a character type n-gram (for example, when the feature is a character n-gram or a dictionary word feature). For example, when the number of characters included in the target string is limited and the length of the feature is smaller than the window width, the number of character n-gram features or dictionary word features corresponding to the characters or strings in the extraction range indicated by the window width information is finite, so a similar pre-score calculation is effective. Furthermore, the above pre-score calculation process may be performed for all character combinations or character type combinations in the extraction range indicated by the window width information, or may be performed for some of the combinations.

[0100] Furthermore, the boundary determination unit 137 can output a score indicating whether or not a word boundary exists between characters to be determined under optimized processing (e.g., memory optimization processing). Here, memory optimization processing includes, for example, processing to optimize the memory usage of an array and processing to optimize the size of information indicating the attributes of an array, which are used in pattern matching processing of feature information included in a weight array and feature information corresponding to a specific character space. Note that these optimization processing methods are applicable not only to pattern matching of feature information but also to pattern matching of character strings more generally.

[0101] Furthermore, the boundary determination unit 137 can perform boundary determination processing between characters in a plurality of divided character strings in parallel.

[0102] FIG. 11 is a diagram showing an example of information stored in the determined character string storage unit 138. The information stored in the determined character string storage unit 138 includes, for example, a determined character string ID and determined character string information. Here, the determined character string ID is determined character string identification information that identifies a character string in which the presence or absence of a word boundary is set between at least one character of the target character string. Furthermore, the determined character string information is information related to a character string in which the presence or absence of a word boundary is set between at least one character of the target character string, and is, for example, information in which information indicating the position of a word boundary (for example, " / ") is added between characters of the target character string. Note that the information indicating the position of a word boundary is not limited to " / ", and other symbols or methods may be used.

[0103] The output unit 141 outputs the determined character string information stored in the determined character string storage unit 138. The output unit 141 may output the determined character string information to the user terminal 112, or may output it to an information processing system that performs natural language processing subsequent to word boundary determination, or may output it to another information processing system.

[0104] 12 is a diagram showing experimental results regarding the processing speed of the natural language processing system 100. The diagram shown in Fig. 12 shows the results of an experiment comparing the processing speed of the natural language processing system 100 with the processing speed of a processing method different from that of the natural language processing system 100. Note that in this experiment, the second weight array calculation unit 131 performed a process of integrating the weights of two or more features corresponding to characters or character strings in an inclusive relationship, but the second weight array calculation unit 131 did not perform a process of integrating the weights of two or more features corresponding to the same character or character string, and the boundary determination unit 137 did not perform a process of calculating a pre-score.

[0105] In the diagram shown in Figure 12, the number of characters processed per second (10 6 Also, "KyTea" (Kyoto Text Analysis Toolkit) shows the results of a different system that uses a different processing method from the natural language processing system 100. "This embodiment" shows the results of processing in the natural language processing system 100, and "This embodiment (4 threads)" shows the results of processing in the natural language processing system 100 that uses parallel processing in four threads.

[0106] 12, the natural language processing system 100 can perform word boundary determination processing at approximately four times the speed of a system different from the natural language processing system 100. Furthermore, the results of another experiment (not shown) confirmed that the natural language processing system 100 in which the second weight array calculation unit 131 integrated the weights of two or more features corresponding to the same character or character string, the natural language processing system 100 in which the boundary determination unit 137 calculated a pre-score, and the natural language processing system 100 in which the second weight array calculation unit 131 integrated the weights of two or more features corresponding to the same or inclusive characters or character strings and the boundary determination unit 137 calculated a pre-score can all perform word boundary determination processing faster than a system different from the natural language processing system 100.

[0107] In the comparative experiment shown in Figure 12, the KyTea distribution model "Compact SVM model" was used as the trained model. The target strings were the "Kyoto Free Translation Task Training Set." Error bars indicate standard deviation. The error rate, which indicates the difference in segmentation positions between "KyTea" and "this embodiment," was 0.016%.

[0108] When the object of the determination process is, for example, a legal document such as a contract, the natural language processing system 100 can further perform article division processing, article type determination processing, and display processing. Note that when the object of the determination process is a document other than a contract, the natural language processing system 100 can perform segmentation processing, type determination processing, and display processing according to the nature of the document. Below, an example will be described in which the object of the determination process of the natural language processing system 100 is a legal document such as a contract.

[0109] Fig. 13 is a diagram showing an example of a contract 1300 that is the target of determination processing by the natural language processing system 100. The contract 1300 shown in Fig. 13 is a confidentiality agreement between two companies, and includes an area 1301 showing the title of the contract, an area 1302 showing a preamble, and areas 1311, 1312, and 1313 showing clauses.

[0110] Note that before the natural language processing system 100 performs the clause division process, clause type determination process, and display process described below, the natural language processing system 100 is unable to recognize that area 1301 indicates the title of the contract, area 1302 indicates the preface to the contract, and areas 1311, 1312, and 1313 indicate clauses.

[0111] The clause dividing unit 142 divides the contract 1300 into sections such as a title, a preface, and clauses. The clause dividing unit 142 can divide the contract 1300 based on, for example, determined character string information stored in the determined character string storage unit 138 following a determination process by the boundary determination unit 137.

[0112] When the clause dividing unit 142 divides the contract 1300 based on the determined character string information, for example, the clauses before and after the words "Article X" can be used as the separators for "clause." The clause dividing unit 142 may divide the contract 1300 by clause, or by paragraph or item.

[0113] If the processing target is a document other than a contract, etc., the clause dividing unit 142 can perform a segmentation process according to the nature of the document. For example, if the processing target is an academic paper, the clause dividing unit 142 can divide the document into segments such as "background," "method," "results," and "discussion."

[0114] The provision type determination unit 143 determines the type of each segment divided by the provision division unit 142. The type is information indicating the category of the written content, and includes "title," "preface," and "article" in the case of a legal document such as a contract, for example. The provision type determination unit 143 can determine, for example, that a part containing the words "Article X" at the beginning of a segment is of the type "article."

[0115] If the processing target is a document other than a contract, etc., the provision type determination unit 143 can perform a delimiter / divider process according to the nature of the document. For example, if the processing target is an academic paper, the provision type determination unit 143 can determine the type of the document, such as "background," "method," "result," or "discussion."

[0116] As a result of the clause division process by the clause division unit 142 and the clause type determination process by the clause type determination unit 143, the natural language processing system 100 can recognize that, for the contract 1300, area 1301 indicates the title of the contract, area 1302 indicates the preface of the contract, and areas 1311, 1312, and 1313 indicate clauses.

[0117] The display unit 144 displays the contract 1300 and the processing results of the clause division unit 142 and clause type determination unit 143 on the user terminal 112 based on the processing results of the clause division unit 142 and clause type determination unit 143 .

[0118] FIG. 14 is a flowchart showing an example of a process for generating a second weight array in the natural language processing system 100.

[0119] First, the natural language processing system 100 acquires training dictionary information and training string information from the information processing system 111 (S1401). Next, the natural language processing system 100 extracts two or more features corresponding to the character spaces in the training string to generate feature information (S1402). Subsequently, the natural language processing system 100 performs training based on the training string information and at least one feature information corresponding to the character spaces in the training string, and generates first weight array information and a trained model (S1403). Note that here, the natural language processing system 100 may also perform training based on the training dictionary information.

[0120] Then, based on the first weight array information, the natural language processing system 100 integrates the weights of two or more features corresponding to at least one pair of identical or inclusive characters or character strings included in the first weight array information to calculate a second weight array (S1404).

[0121] FIG. 15 is a flowchart showing an example of a process for determining word boundaries in the natural language processing system 100.

[0122] First, the natural language processing system 100 acquires target string information for determining word boundaries based on the user terminal 112 or a user instruction via the user terminal 112 (S1501). Next, the natural language processing system 100 divides the target string to generate divided string information (S1502). Next, the natural language processing system 100 extracts, for each divided string, two or more features corresponding to the spaces between characters in the divided string (S1503).

[0123] Next, the natural language processing system 100 determines, for each segmented string, whether there is a word boundary between the characters of the segmented string based on the second weight array information and two or more pieces of feature information corresponding to the spaces between the characters of the extracted segmented string (S1504). At this time, the natural language processing system 100 may determine the presence or absence of word boundaries based on scores regarding the presence or absence of word boundaries in all or some combinations of characters or all or some combinations of character types in the extracted range, calculated based on the window width information. Thereafter, the natural language processing system 100 outputs determined string information regarding determined strings for which the presence or absence of word boundaries has been set in the target string based on the determination result (S1505).

[0124] Furthermore, the natural language processing system 100 divides the article and determines the article type based on the determined character string information (S1506).Then, the natural language processing system 100 displays the processing results of dividing the article and determining the article type on the user terminal 112 (S1507).

[0125] The above describes one embodiment of the present invention. The natural language processing system 100 can determine whether or not there is a word boundary between characters in a target string based on a second weight array calculated by combining in advance at least one set of two or more feature weights corresponding to identical or inclusive characters or strings included in a first weight array. This reduces data volume and enables word boundary determination at a higher processing speed.

[0126] Furthermore, the natural language processing system 100 can determine the presence or absence of word boundaries between characters corresponding to the at least one extracted feature based on a score regarding the presence or absence of word boundaries for all or some combinations of characters or all or some combinations of character types in the extracted range, calculated based on window width information, thereby enabling word boundary determination to be performed at a higher processing speed.

[0127] Furthermore, the natural language processing system 100 can generate a second weight array by adding or averaging weights of at least one set of two or more features corresponding to identical or inclusive characters or character strings based on the first weight array. This allows the presence or absence of word boundaries to be determined at higher processing speed using a weight array with reduced data volume.

[0128] Furthermore, the natural language processing system 100 can determine whether or not there is a word boundary based on at least one feature corresponding to a space between characters in a target string or a segmented string, and the second weight array. This makes it possible to determine whether or not there is a word boundary between the characters to be determined by evaluating only the character strings within a certain range before and after the space between the characters to be determined, without evaluating the entire target string.

[0129] Furthermore, the natural language processing system 100 can divide a target string into two or more divided strings, each containing two or more characters, and determine for each divided string whether there is a word boundary between the characters in the divided string based on at least one feature corresponding to the space between the characters in each divided string and the second weight array. This allows word boundary determination to be processed in parallel for one target string, thereby enabling word boundary determination of the target string to be performed at a higher processing speed.

[0130] Furthermore, based on the determination result, the natural language processing system 100 can output determined character string information regarding a determined character string in which a word boundary is set between at least one character of the target character string. This allows the natural language processing system 100 or an external information processing system to perform natural language processing based on the target character string in which word boundaries are set, and also allows the client to obtain the target character string in which word boundaries are set.

[0131] It should be noted that the present embodiment is provided to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0132] The processes and procedures according to the present disclosure may be realized not only by the devices explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described herein may be realized by implementing logic corresponding to the processes in a medium such as an integrated circuit, volatile memory, non-volatile memory, a magnetic disk, or optical storage. Furthermore, the processes and procedures described herein may be implemented as computer programs and executed by various computers, including terminal devices and server devices.

[0133] Processes and procedures described herein as being performed by a single device, software, component, and / or module may be performed by multiple devices, multiple software, multiple components, and / or multiple modules. Furthermore, various pieces of information described herein as being stored in a single memory and storage device may be stored in multiple memories included in a single device or in multiple memories distributed across multiple devices. Furthermore, the software and hardware described herein may be realized by integrating them into fewer components or by decomposing them into more components. [Explanation of symbols]

[0134] 100 natural language processing system, 111 information processing system, 112 user terminal, 121 learning dictionary acquisition unit, 122 learning dictionary storage unit, 123 learning string acquisition unit, 124 learning string storage unit, 125 feature extraction unit, 126 learning unit, 127 first weight array storage unit, 128 trained model storage unit, 131 second weight array calculation unit, 132 second weight array storage unit, 133 target string acquisition unit, 134 target string storage unit, 135 string division unit, 136 divided string storage unit, 137 boundary determination unit, 138 string storage unit, 141 output unit, 142 article division unit, 143 article type determination unit, 144 display unit

Claims

1. a second weight array calculation unit that calculates a second weight array by integrating weights of at least one set of two or more features corresponding to the same or inclusive characters or character strings, based on a first weight array relating to weights for determining word boundaries of all or part of the two or more features generated by training using at least one training string in which the presence or absence of a word boundary is set between at least one character and two or more features corresponding to any character between the characters in the training string; a feature extraction unit that extracts at least one feature corresponding to a space between characters in a target string including two or more characters based on window width information regarding a preset extraction range; a boundary determination unit that determines whether or not there is a word boundary between characters in the target string that corresponds to the at least one extracted feature, based on the trained model generated by the training, the second weight array, and the at least one extracted feature; A natural language processing system comprising:

2. 2. The natural language processing system according to claim 1, wherein the boundary determination unit determines whether or not there is a word boundary between characters corresponding to the at least one extracted feature based on a score regarding the presence or absence of a word boundary in a combination of all or a part of characters or a combination of all or a part of character types in the extraction range, the score being calculated based on the window width information.

3. a character string dividing unit that divides the target character string into two or more divided character strings, each including two or more characters; the feature extraction unit extracts, for each of the divided character strings, at least one feature corresponding to a space between characters in the divided character strings; the boundary determination unit determines, for each of the divided character strings, whether or not there is a word boundary between characters corresponding to the at least one extracted feature in the divided character string, based on the trained model, the second weight array, and the at least one extracted feature; 3. The natural language processing system according to claim 1.

4. The natural language processing system according to any one of claims 1 to 3, wherein the second weight array calculation unit calculates the second weight array by adding weights of the two or more features corresponding to characters or character strings that are identical or have an inclusive relationship based on the first weight array.

5. The natural language processing system according to any one of claims 1 to 3, wherein the second weight array calculation unit calculates the second weight array by averaging weights of the two or more features corresponding to identical or inclusive characters or character strings based on the first weight array.

6. the two or more features corresponding to character spaces in the training string are characters or character strings between a predetermined character space in the training string and a predetermined relative position starting from the predetermined character space, the feature corresponding to a character space in the target character string extracted by the feature extraction unit is a character or character string between a predetermined character space in the target character string and a predetermined relative position starting from the predetermined character space; The natural language processing system according to any one of claims 1 to 5.

7. the two or more features corresponding to character spaces in the training string are character types of characters or character types of strings between a predetermined character space in the training string and a predetermined relative position starting from the predetermined character space, the feature corresponding to a character space in the target character string extracted by the feature extraction unit is a character type of a character or a character type of a character string between a predetermined character space in the target character string and a predetermined relative position starting from the predetermined character space; The natural language processing system according to any one of claims 1 to 5.

8. the first weight array is generated by learning using a learning dictionary including at least one word, at least one learning string in which the presence or absence of a word boundary is set between at least one character, and two or more features corresponding to any character intervals in the learning string; the two or more features corresponding to a character space in the training string are the at least one word included in the training dictionary, among characters or character strings located between a predetermined character space in the training string and a predetermined relative position starting from the predetermined character space; the feature corresponding to a character space in the target string extracted by the feature extraction unit is the at least one word included in the learning dictionary, among characters or character strings located between a predetermined character space in the target string and a predetermined relative position starting from the predetermined character space; The natural language processing system according to any one of claims 1 to 5.

9. The natural language processing system according to any one of claims 1 to 8, further comprising an output unit that outputs determined character string information regarding a determined character string in which a word boundary is set between at least one character of the target character string based on the determination result by the boundary determination unit.

10. The computer a first weight array relating to weights for determining word boundaries of a string for all or part of two or more features generated by training using at least one training string in which the presence or absence of a word boundary is set between at least one character and two or more features corresponding to any character between the characters in the training string; and a second weight array calculated by integrating weights of at least one set of two or more features corresponding to the same or inclusive characters or strings; extracting at least one feature corresponding to a character space between characters in a target character string including two or more characters based on window width information relating to a preset extraction range; determining whether or not there is a word boundary between characters corresponding to the at least one extracted feature in the target string based on the trained model generated by the training, the second weight array, and the at least one extracted feature; Natural language processing methods.

11. On the computer, a second weight array calculation unit that calculates a second weight array by integrating weights of at least one set of two or more features corresponding to the same or inclusive characters or character strings, based on a first weight array relating to weights for determining word boundaries of all or part of the two or more features generated by training using at least one training string in which the presence or absence of a word boundary is set between at least one character and two or more features corresponding to any character between the characters in the training string; a feature extraction unit that extracts at least one feature corresponding to a space between characters in a target string including two or more characters based on window width information regarding a preset extraction range; a boundary determination unit that determines whether or not there is a word boundary between characters in the target string that corresponds to the at least one extracted feature, based on the trained model generated by the training, the second weight array, and the at least one extracted feature; A natural language processing program to achieve this.

Images