Information processing program, information processing method, and information processing device

Abstract

To enable appropriate adjustment of a tokenizer which inputs data into a machine learning model regardless of the machine learning model.SOLUTION: An information processing program causes a computer to execute processing for: training a tokenizer 11 on the basis of a bias in an appearance degree in a plurality of classes for each of character strings included in a plurality of divided character string data obtained by dividing character string data by a plurality of different division patterns corresponding to the character string data obtained by inputting the character string data into the tokenizer 11 when training the tokenizer 11 by using a training dataset including a plurality of training data associating the character string data with class information indicative of a class to which the character string data belongs among the plurality of classes. The tokenizer 11 outputs divided character string data serving as input data input into a machine learning model 12 and outputs the divided character string data including the plurality of character strings by dividing the character string data to be input.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] In natural language processing using a machine learning model, character string data such as text that is input to the machine learning model may be divided into smaller units, such as tokens. A token is one or more character strings (phrases, words) obtained by dividing the character string data at one or more positions.

[0003] For example, if the text is "I don't hate the taste," splitting it in a certain way will give the tokens "taste," "hate," "ja," and "nai." Also, if the text is "_not_bad_taste" (the underscore (_) represents a single space), splitting it in a certain way will give the tokens "_not," "_bad," "_tas," and "te."

[0004] Hereinafter, for convenience, tokens may be referred to as "words." "Words" may include, for example, general words such as "hate" and "_not," as well as subwords such as "ja," "_tas," and "te." The process of dividing character string data into tokens may be called "word division" or "tokenization."

[0005] The way text is divided in word segmentation, in other words, the segmentation pattern, can affect the performance of natural language processing using a machine learning model, for example, the accuracy of the inference results output from the machine learning model.

[0006] For example, word segmentation by a word segmenter (lexical segmenter) may be performed as pre-processing, and the segmented text obtained by word segmentation may be used in downstream processing (downstream task). In downstream processing, for example, inference processing is performed using a trained machine learning model (hereinafter sometimes referred to as the "downstream model") using the segmented text as input data. In such processing, by changing the segmentation pattern of word segmentation, it may be possible to improve performance in natural language processing without having to adjust the parameters of the downstream model.

[0007] For example, a method is known in which a word segmenter and a subsequent model are started in parallel to simultaneously optimize (adjust) the word segmentation and the subsequent model. In this method, for example, input data containing multiple texts is input to the word segmenter, and the resulting segmented text is input to the subsequent model, and the training results of the subsequent model are fed back to the word segmenter, thereby optimizing the word segmenter. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2022-077202 Summary of the Invention [Problem to be solved by the invention]

[0009] In the above-mentioned method, the subsequent model is fixed, for example, the type, parameters, architecture, etc. of the subsequent model are fixed, so while the word segmenter can be adjusted to optimize it for the subsequent model, its usage is limited to situations where the subsequent model is fixed.

[0010] In one aspect, an object of the present invention is to enable a lexical segmenter that inputs data to a machine learning model to be appropriately adjusted independently of the machine learning model. [Means for solving the problem]

[0011] In one aspect, the information processing program may cause a computer to execute the following process: The process may include training a lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, the lexical segmenter outputting the segmented character string data containing a plurality of character strings by segmenting input character string data, using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, based on bias in the frequency of occurrence of each character string in the plurality of classes, for each character string included in the plurality of segmented character string data obtained by segmenting the character string data using a plurality of different segmentation patterns that correspond to the character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs. [Effects of the Invention]

[0012] In one aspect, the present invention can make it possible to appropriately adjust a lexical segmenter that inputs data to a machine learning model, regardless of the machine learning model. [Brief explanation of the drawings]

[0013] [Figure 1] 10A and 10B are diagrams illustrating examples of changes in the inference results of the second-stage model according to the word segmentation pattern. [Figure 2] FIG. 10 is a diagram for explaining a technique according to an embodiment. [Figure 3] FIG. 10 is a diagram for explaining a detailed example of a score adjustment technique. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer that realizes the functions of a server according to an embodiment. [Figure 5] FIG. 2 is a block diagram illustrating an example of a software configuration of a server according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of parameters. [Figure 7]FIG. 10 is a diagram illustrating an example of updated parameters. [Figure 8] FIG. 1 is a diagram illustrating an example of a training dataset. [Figure 9] FIG. 10 is a diagram for explaining an example of training a word segmenter according to the first technique. [Figure 10] FIG. 10 is a diagram for explaining an example of training a word segmenter according to the second technique. [Figure 11] 10A and 10B are diagrams illustrating examples of document classification results according to word segmenter training methods. [Figure 12] FIG. 10 is a diagram showing an example of document classification results according to the training method of the word segmenter when pseudo labels are used. [Figure 13] 10 is a flowchart illustrating an example of an operation of a process for training a word segmenter using a first technique in a server according to an embodiment. [Figure 14] 10 is a flowchart illustrating an example of an operation of a process for training a word segmenter using a second technique in a server according to an embodiment. [Figure 15] 10 is a flowchart illustrating an example of the operation of an inference process by a server according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments will be described with reference to the drawings. However, the embodiments shown below are merely examples, and are not intended to exclude various modifications or application of techniques not explicitly stated in the embodiments. For example, the present embodiment can be implemented with various modifications within the scope of the spirit thereof. Furthermore, each figure does not intend to include only the components shown in the figure, but may include other functions, etc.

[0015] [A] Explanation of word segmentation adjustment First, we will explain how to adjust word segmentation. A word segmenter (lexical segmenter) performs word segmentation on input text, and inputs the resulting multiple words to a subsequent model. An example of a word segmenter is a tokenizer. Text is an example of character string data. Furthermore, multiple words (tokens) segmented from text by a word segmenter are an example of segmented character string data. Hereinafter, multiple words (tokens) may be referred to as "segmented text."

[0016] The latter-stage model is a trained machine learning model that executes various AI (Artificial Intelligence) tasks related to natural language processing. The latter-stage model may be, for example, a document classifier that performs document classification of input text. Document classification is a process of analyzing the latent meaning of input text and classifying the text into one of multiple classes. Examples of document classification include various document classification tasks such as sentiment classification and topic classification. Note that the AI ​​task executed by the latter-stage model is not limited to document classification, and may include various inference processes that use text as input data and output inference results.

[0017] 1 is a diagram showing an example of how the inference result of the second-stage model changes depending on the word segmentation pattern. As a first-stage process (preprocessing), the word segmenter 110 segments the input text into multiple words (tokens) by word segmentation.

[0018] Figure 1 shows an example in which the word splitter 110 splits each text into words using the following two splitting patterns when two texts, "I don't hate the taste" (_not_bad_taste) and "I hate the taste" (_bad_taste), are input to the word splitter 110. "I don't like the taste" "I don't hate the taste" "I don't like the taste" (_bad / _taste) "I don't like the taste" (_bad / _tas / te)

[0019] Hereafter, as in the above splitting pattern, text divisions due to word splitting are represented by a slash ( / ). In other words, each character string from the beginning of the text to the first slash, between the slashes, and from the last slash to the end of the text is a word (token).

[0020] It should be noted that the technique according to one embodiment is applicable to various languages. In the above-described example, the parentheses attached to the text indicate, for convenience, an example of the meaning of the text in a second language (English in the example of FIG. 1) different from the first language of the text (Japanese in the example of FIG. 1). In the notation in the second language, an underscore (_) indicates a half-width space as a single character. Furthermore, the parentheses attached to multiple words (tokens) indicate, for convenience, the meaning of the words obtained by word segmentation in a second language (e.g., English) different from the first language of the words (e.g., Japanese).

[0021] Also, tokens do not necessarily have to be meaningful units, such as when one word is divided into multiple subwords. Furthermore, because the language structure, such as word order and the presence or absence of "word segments," may differ between the first and second languages, words at the same delimiter position (e.g., "taste" and "_not") do not necessarily correspond between the first and second languages. In other words, the text and words in the first language and the text and words in the second language written in parentheses can be considered to be independent of each other between the first and second languages.

[0022] The document classifier 120 is an example of a later-stage model that performs later-stage processing. The document classifier 120 according to one embodiment receives input data, which is segmented text obtained by word segmentation of text by the word segmenter 110, and classifies the segmented text into one of a plurality of classes. For example, the document classifier 120 performs sentiment classification as an example of document classification. Sentiment classification is a task of analyzing the latent meaning of the input segmented text and classifying the segmented text into a plurality of classes, such as emotional classes such as "POSITIVE" and "NEGATIVE." Note that, although the following description illustrates an example in which the number of classes is two, the number of classes may be three or more.

[0023] In the example of Figure 1, "POSITIVE" is an appropriate class for "the taste is not bad" (_not_bad_taste). For example, the document classifier 120 may classify the segmented text "the taste is / not / bad" (_not_bad / _tas / te) as POSITIVE due to the influence of the word "not_bad" shown in a square. On the other hand, the document classifier 120 may classify the segmented text "the taste is / not / bad / _tas / te) as NEGATIVE due to the influence of the word "bad" in bold.

[0024] As such, different segmentation patterns for the same text will result in different sentiment classification results (inference results), so it is important to perform appropriate word segmentation that will improve the performance of subsequent processing, for example, word segmentation that can correctly predict document classification classes.

[0025] For example, by using a method that simultaneously optimizes (adjusts) the word segmentation and the latter-stage model by running the word segmenter and the latter-stage model in parallel, the word segmenter can be adjusted to optimize for the latter-stage model. However, this method limits its use to situations where the latter-stage model is already determined, which can reduce the versatility of the word segmenter. Thus, there is a trade-off between improving the inference accuracy of the latter-stage model and the versatility of the word segmenter.

[0026] In one aspect, an embodiment describes a technique for enabling a word segmenter that inputs data to a machine learning model to be appropriately adjusted without relying on the machine learning model. For example, according to the technique of one embodiment, it is possible to train a word segmenter that contributes to improving the performance of a subsequent model in a usage scenario in which the subsequent model has not been determined. A usage scenario in which the subsequent model has not been determined is, for example, a situation in which the combination of text and classes used in subsequent processing is given, but the subsequent model, its parameters, and the architecture to be used for classification have not been determined. In another aspect, an embodiment describes a technique for improving the versatility of the word segmenter while maintaining performance at or above the trade-off level for the inference accuracy of the subsequent model.

[0027] [B] Description of a method according to one embodiment FIG. 2 is a diagram illustrating a technique according to one embodiment. The word segmenter 11 is an example of a lexical segmenter that, as a first-stage process, segments input string data to output segmented string data containing multiple strings. The document classifier 12 is an example of a second-stage model (machine learning model) that, as a second-stage process, performs inference processing using the segmented string data output from the word segmenter 11. In other words, the segmented string data output from the word segmenter 11 becomes input data to the document classifier 12. The word segmenter 11 and the document classifier 12 may each be similar to the word segmenter 110 and the document classifier 120 shown in FIG. 1, except for the parts described below.

[0028] First, with reference to symbol A in Fig. 2, an example of a method for outputting segmented text by the word segmenter 11 that performs the preceding processing will be described. In symbol A (symbols A1 and A2), multiple words (tokens) obtained by word segmenting text using multiple segmentation patterns are shown in rounded rectangular boxes. In the rounded rectangular boxes, words obtained by word segmentation are shown in angle brackets, and the scores of the words (token scores) are shown below the words. Note that <sentence start> or<BOS(Beginning of Sentence)> is the beginning of the text, and <end of the text> or<EOS(End of Sentence)> is the end of the sentence in the text. The route from the beginning to the end of the sentence, via the rounded rectangular boxes and the solid lines connecting the rounded rectangular boxes, is a "path," which is an example of a division pattern.

[0029] As indicated by the reference symbols A1 and A2, the word segmenter 11 selects as output a path that maximizes the sum of the scores of the multiple words (tokens) obtained by segmenting the text.

[0030] In the example of code A1, the word segmenter 11 selects the segmentation pattern "taste is / kiranai ja / nai" with the maximum total score (1.21) as the segmented text of the text "taste is not / kiranai" from the following results. "I don't dislike the taste": 0.30 + 0.20 + 0.43 = 0.93 "I don't dislike the taste": 0.30 + 0.20 + 0.15 + 0.19 = 0.84 "I don't dislike the taste": 0.30 + 0.72 + 0.19 = 1.21

[0031] In the example of code A2, the word segmenter 11 selects the segmentation pattern "_not_bad / _tas / te" with the maximum total score (1.06) as the segmented text of the text "_not_bad_taste" from the following results. ·「_not / _bad / _taste」:0.30+0.20+0.43=0.93 ·“_not / _bad / _tas / te”:0.30+0.20+0.15+0.19=0.84 ·「_not_bad / _taste」:0.72+0.23=0.95 ·“_not_bad / _tas / te”:0.72+0.15+0.19=1.06

[0032] The combination of a word and its score is held as a parameter in the word segmenter 11 and is referenced for selecting a selection pattern when word segmentation is performed.

[0033] In one embodiment, the word score is adjusted so that the word segmenter 11 performs word segmentation using a specific segmentation pattern, thereby outputting appropriate segmented text from the word segmenter 11. Appropriate segmented text is, for example, data that, when input as input data to a subsequent model, will produce appropriate inference results in subsequent processing by the subsequent model. In the example of FIG. 2, appropriate segmented text is segmented text such that "I don't hate the taste" or "_not_bad_taste" is sentiment-classified by the document classifier 12 into an appropriate class (e.g., POSITIVE), and is, for example, a plurality of words including the words "I hate the taste" or "_not_bad" shown in a shaded rounded square. That is, in one embodiment, as shown by reference numerals A1 and A2, the score is adjusted so that the word segmenter 11 outputs the path indicated by the thick solid line.

[0034] Next, a score adjustment method according to one embodiment will be described with reference to symbol B in Fig. 2. The adjustment method according to one embodiment adjusts the score based on the bias in the appearance frequency of each word included in the segmented text among multiple classes.

[0035] Graph B1 is an example of a score that indicates the bias in the frequency of occurrence (degree of occurrence) of each word (token) class. The vertical axis of graph B1 indicates the token score e(s), and the horizontal axis indicates the bias in the frequency of occurrence of the token for each class p(s|c). Here, for a certain text S, let V(S) denote the set of word segmentation candidates S' for text S that can be created based on a vocabulary V consisting of |V| tokens. s is one of the |S'| tokens that make up the word segmentation candidate S', and S'= <s1, s2, ..., s |s'| > where c is one of the classes POSITIVE, NEGATIVE, etc.

[0036] Note that symbol B shows a graph in a two-dimensional space where the number of classes is two, but if the number of classes is three or more, a graph B1 in a three-dimensional space including a horizontal axis for each combination of two classes may be generated, and the score e(s) of each token may be obtained based on the graph B1.

[0037] e(s) indicates the score of token s according to p(s|c). p(s|c) indicates the probability that token s is included (appears) in text of class c (for example, the probability that "I hate you" is included in text whose class is POSITIVE). For example, p(s|c) = 0.5 means that token s appears evenly in multiple classes c, while p(s|c) = 0.0 or 1.0 means that token s appears only in a specific class c. In the example in Figure 2, the score of token s that appears only in POSITIVE text is p(s|c = "POSITIVE") ≒ 1.0, and the score of token s that appears only in NEGATIVE text is p(s|c = "NEGATIVE") = 0.0.

[0038] Graph B1 is represented by a function such that, for example, the score e(s) of a token s that appears evenly in multiple classes c is minimum, and the score e(s) of a token s that appears only in a specific class among the multiple classes c is maximum.

[0039] 3 is a diagram for explaining a detailed example of a score adjustment method. As indicated by the symbol C, the score e(s) is calculated (scored) by using the frequency of occurrence of token s for each class c as entropy, for example, according to the following formula (1):

number

[0040] In the above formula (1), c∈C is a set C of class c, for example {POSITIVE, NEGATIVE}. |C| is the total number of classes included in set C, for example, 2 (POSITIVE, NEGATIVE).

[0041] As shown by symbol D, the score e(s) calculated by the above method is applied to the final score p'(s) of token s. The final score p'(s) is calculated, for example, according to the following formula (2):

number

[0042] In the above formula (2), p(s) is the probability that token s appears in the entire data, and is an example of the final score for each token s used, for example, in the word segmenter 110 (see FIG. 1). μ is a predetermined weight. q(s) is a function for applying information on the frequency of occurrence of token s for each class c as a weight to the final score p'(s) based on the above score e(s), and is calculated, for example, according to the following formula (3).

number

[0043] In the above formula (3), ε is a small value that prevents the probability of token s from becoming 0. V is a set (a list with no duplicates) of all possible tokens (vocabulary), and t is token t included in the set (vocabulary) V. Σ in the denominator on the right-hand side means dividing the numerator by the scores of all possible tokens t so that the sum is 1.

[0044] The final score p'(s) is used to determine the path to be selected as the output in the word segmenter 11, as indicated by the symbol F (F1 and F2).

[0045] Thus, according to a technique of one embodiment, the parameters of word segmenter 11 are adjusted so that the score of a token s with a greater bias in the frequency of occurrence among multiple classes is increased. Then, for each of multiple input texts, word segmenter 11, based on the parameters, outputs the segmented text that maximizes the sum of the scores of multiple words from among the multiple segmented texts obtained by inputting the text into word segmenter 11. Therefore, by training word segmenter 11 based on class information, word segmenter 11 can be optimized to output input data that enables document classifier 12 to perform appropriate document classification.

[0046] As described above, in one embodiment, word segmentation is performed that incorporates information about the class c∈C used in document classification by the document classifier 12. That is, in one embodiment, to achieve word segmentation that takes class information into consideration, the score p(s), which is the token probability used by the word segmenter 110, is replaced with a weighted sum using μ of the score p(s) and a score p'(s) based on q(s)(e(s)) that includes information about the class c∈C, as shown in the above formula (2).

[0047] [C] Server configuration example Next, a configuration example of the server 1 (see FIG. 5) that realizes the method according to the embodiment described above will be described.

[0048] [C-1] Hardware configuration example The server 1 according to one embodiment may be a virtual server (VM: Virtual Machine) or a physical server. The functions of the server 1 may be implemented by one computer or two or more computers. Furthermore, at least some of the functions of the server 1 may be implemented using hardware (HW) resources and network (NW) resources provided by a cloud environment.

[0049] 4 is a block diagram showing an example of a hardware (HW) configuration of a computer 10 that realizes the functions of the server 1 as an example of an embodiment. When multiple computers are used as HW resources that realize the functions of the server 1, each computer may have the HW configuration shown in FIG.

[0050] As shown in FIG. 4, the computer 10 may, as a hardware configuration, illustratively include a processor 10a, a graphics processing unit 10b, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a reading unit 10g.

[0051] The processor 10a is an example of a processing unit that performs various controls and calculations. The processor 10a may be connected to each block in the computer 10 via a bus 10j so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multi-core processor having multiple processor cores, or a configuration having multiple multi-core processors.

[0052] The processor 10a may be, for example, an integrated circuit (IC) such as a CPU, MPU, APU, DSP, ASIC, or FPGA. Note that the processor 10a may be a combination of two or more of these integrated circuits. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, APU is an abbreviation for Accelerated Processing Unit, DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0053] The graphics processing device 10b controls screen display for an output device such as a monitor in the IO unit 10f. The graphics processing device 10b may also be configured as an accelerator that executes machine learning processing and inference processing using a machine learning model. The graphics processing device 10b may be various types of arithmetic processing devices, such as a GPU (Graphics Processing Unit), an APU, a DSP, an ASIC, an FPGA, or other integrated circuits (ICs).

[0054] The memory 10c stores various types of data, programs, and other information. Examples of the memory 10c include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0055] The storage unit 10d stores various types of data, programs, and other information. Examples of the storage unit 10d include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of nonvolatile memory include a flash memory, a storage class memory (SCM), and a read-only memory (ROM).

[0056] The storage unit 10d may store a program 10h (information processing program) that realizes all or part of the various functions of the computer 10. For example, the processor 10a of the server 1 can realize the function of a control unit 30 (see FIG. 5), which will be described later, by expanding the program 10h stored in the storage unit 10d into the memory 10c and executing it.

[0057] The IF unit 10e is an example of a communication IF that controls the connection and communication between the server 1 and other computers. For example, the IF unit 10e may include an adapter that complies with electrical communications (e.g., LAN (Local Area Network)) such as Ethernet (registered trademark) or optical communications such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods. The program 10h may be downloaded to the computer 10 from a network via the communication IF and stored in the storage unit 10d.

[0058] The IO unit 10f may include one or both of an input device and an output device. Examples of input devices include a keyboard, a mouse, and a touch panel. Examples of output devices include a monitor, a projector, and a printer. The IO unit 10f may also include a touch panel that combines an input device and a display device. The output device may be connected to the graphics processing device 10b.

[0059] The reading unit 10g is an example of a reader that reads data and program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 10i may store the program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the memory unit 10d.

[0060] Examples of the recording medium 10i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

[0061] The above-described HW configuration of the computer 10 is an example. Therefore, the HW in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or the HW may be added or deleted as needed.

[0062] [C-2] Software configuration example 5 is a block diagram showing an example of the software configuration of the server 1 as an example of an embodiment. The server 1 is an example of a computer or information processing device that trains the word segmenter 11 by adjusting the score.

[0063] 5, the server 1 may illustratively include a word segmenter 11, a document classifier 12, a segmentation control unit 31, and an inference control unit 32. The server 1 may further include a pseudo-class generation unit 33. The blocks 31 to 33 included in the server 1 are an example of the control unit 30. The functions of the control unit 30 may be realized, for example, by the processor 10a of the computer 10 shown in FIG. 4 executing a program 10h deployed in the memory 10c.

[0064] 5, the server 1 may exemplarily include a storage area capable of storing word information 11a, score information 11b and 11c, a training dataset 21, an inference dataset 22, a token set 23, and an inference result 24. In addition to the above, the server 1 may also include a storage area capable of storing data of the word splitter 11 and the document classifier 12, for example, model data such as a neural network.

[0065] The storage area of ​​the server 1 may be realized by, for example, one or both of the storage areas of the memory 10c and the storage unit 10d of the computer 10 shown in FIG.

[0066] The server 1 (control unit 30) may, for example, receive at least one of the training dataset 21 and the inference dataset 22 from another computer (not shown) via the IF unit 10e and the network, and store it in a storage area.

[0067] The server 1 (control unit 30) may also transmit, for example, at least one of the word segmenter 11, score information 11c, and inference result 24 stored in the memory area to another computer (not shown) via the IF unit 10e and the network. Furthermore, the server 1 may also transmit intermediate data, for example, at least one of the word segmenter 11 in the middle of training, word information 11a, score information 11b, and token set 23 to another computer (not shown) via the IF unit 10e and the network.

[0068] When the server 1 transmits either or both of the word segmenter 11 and the score information 11b to another computer (not shown), the functions of the document classifier 12 and the inference control unit 32 may be provided in the other computer instead of the server 1. In this case, the data of the document classifier 12, the inference dataset 22, the token set 23, and the inference result 24 may be stored in a memory area provided in the other computer instead of in a memory area provided in the server 1.

[0069] The word segmenter 11 is an example of the word segmenter 11 shown in FIG. 2. The word segmenter 11 may be a word segmenter that employs, for example, a technique using a unigram language model. For example, the word segmenter 11 may refer to word information 11a indicating vocabulary that the word segmenter 11 can handle, i.e., words (tokens), and score information 11b or 11c that stores scores associated with the vocabulary, and perform word segmentation based on the vocabulary and scores included in the word information 11a and the score information 11b or 11c. The word information 11a and the score information 11b and 11c are examples of parameters of the word segmenter 11.

[0070] Fig. 6 is a diagram showing an example of parameters. Fig. 6 shows an example of word information 11a, which is a list of words (tokens) such as "taste" and "_taste," and score information 11b that has not been updated by the division control unit 31. The score information 11b includes a score p(s) calculated for each word in the word information 11a.

[0071] Fig. 7 is a diagram showing an example of updated parameters. Fig. 7 shows an example of word information 11a and score information 11c updated from score information 11b by segmentation control unit 31. Score information 11c includes score p(s) similar to that in Fig. 6 and score p'(s) (see formula (2) above) calculated based on information related to the class. Note that score information 11c may include score p'(s) without score p(s), or may include score q(s) (see formula (3) above) instead of score p'(s).

[0072] The segmentation control unit 31 trains the word segmenter 11 based on the training dataset 21. For example, when training the word segmenter 11 using the training dataset 21, the segmentation control unit 31 trains the word segmenter 11 based on the bias in the frequency of occurrence in multiple classes for each word (character string) included in multiple segmented texts obtained by inputting text 21a to the word segmenter 11 and segmenting the text 21a using multiple different segmentation patterns corresponding to the text 21a. For example, the segmentation control unit 31 trains the word segmenter 11 based on the bias in the frequency of occurrence in multiple classes for each word (character string) included in the multiple words (segmented texts) for each of the multiple segmented texts.

[0073] Fig. 8 is a diagram showing an example of the training dataset 21. As shown in Fig. 8, the training dataset 21 may include, for example, a plurality of training data sets in which text 21a, which is an example of character string data, is associated with class information 21b indicating to which class c of a plurality of classes (set C) the text 21a belongs. The training data may include, for example, combinations of text and classes used in inference processing by the document classifier 12.

[0074] Training of the word segmenter 11 by the segmentation control unit 31 may be realized by the following first method or second method, depending on whether the word segmenter 11 has undergone machine learning (training) or not.

[0075] (First method) The first technique is a technique of applying the score q(s) to an already machine-learned (trained) word segmenter 11. In the first technique, training of the word segmenter 11 may include a process of updating (adjusting) score information 11b or 11c for multiple words in the parameters of the word segmenter 11. In this case, re-machine learning (retraining) of the word segmenter 11 may be omitted in the training of the word segmenter 11.

[0076] Fig. 9 is a diagram for explaining an example of training of the word segmenter 11 by the first method. Fig. 9 explains the machine learning phase of the word segmenter 11 and the score update phase (training process of the word segmenter 11 by the segmentation control unit 31) in that order.

[0077] 9, in the machine learning phase of the word segmenter 11, the word segmenter 11 segments (word segments) each of the texts 21a included in the training dataset 21 into a plurality of words (tokens) based on the word information 11a. In process P11, the segmentation control unit 31 estimates a score p(s) for each of the plurality of words obtained by segmentation using the frequency of occurrence of the word, and stores the score in score information 11b in association with the word.

[0078] As an example, in the word segmentation process by the word segmenter 11 using a unigram language model, the word segmenter 11 may calculate a word segmentation candidate S'^ that maximizes the word segmentation probability p(S'|S) using a method such as the Viterbi algorithm. For example, the word segmentation candidate S'^ and the probability p(S'|S) may be calculated according to the following equations (4) and (5).

number

[0079] In the above formula (5), the unigram probability p(s) may be estimated using machine learning using the EM (Expectation Maximization) algorithm. Note that the Viterbi algorithm and the EM algorithm are machine learning methods for the word segmenter 11, and are unsupervised learning methods for estimating p(s) and p(s|c).

[0080] It should be noted that the server 1 does not necessarily need to execute the processing of the machine learning phase of the word segmenter 11. For example, the server 1 may acquire the word segmenter 11 that has undergone machine learning and parameters (word information 11a and score information 11b) from another computer.

[0081] 9, in the score update phase, the segmentation control unit 31 calculates the probability p(s|c) for all possible tokens s based on the score information 11b and the multiple training data (text 21a and class information 21b) included in the training dataset 21. Symbol G shows an example of the appearance probability p(s|c) of a word (token) for each class. As shown in symbol G, for each word and for each class, the probability p(s|c=POS) that the word appears in the POSITIVE class and the probability p(s|c=NEG) that the word appears in the NEGATIVE class are calculated.

[0082] The segmentation control unit 31 calculates the score e(s) based on the probability p(s|c) according to the above formula (1), and calculates the score q(s) based on the score e(s) according to the above formula (3). Then, the segmentation control unit 31 estimates the score p'(s) based on the scores p(s) and q(s) according to the above formula (2), and stores it in the score information 11c in association with the word. This allows the word segmenter 11 to perform word segmentation based on the updated parameters (score information 11c) in preprocessing of the inference process of the document classifier 12.

[0083] (Second method) The second technique is a technique for creating word segmenter 11 while taking score q(s) into consideration, and applying score p(s) to the machine learning process of word segmenter 11. In the second technique, training word segmenter 11 may include machine learning process of word segmenter 11 and a process for updating (adjusting) score information 11b or 11c for multiple words in the parameters of word segmenter 11 in the machine learning process.

[0084] 10 is a diagram illustrating an example of training the word segmenter 11 using the second method. As shown in Fig. 10, in process P21, the segmentation control unit 31 inputs text 21a included in the training dataset 21 to the word segmenter 11, and estimates a score p(s) for each of a plurality of words segmented by the word segmenter 11 using the frequency of occurrence of the word.

[0085] The division control unit 31 also calculates the probability p(s|c) for every possible token s based on the estimated score p(s) and multiple pieces of training data (text 21a and class information 21b) included in the training dataset 21. Furthermore, in process P21, the division control unit 31 calculates the score e(s) based on the probability p(s|c) in accordance with the above formula (1), and calculates the score q(s) based on the score e(s) in accordance with the above formula (3).

[0086] Then, in process P21, the segmentation control unit 31 estimates the score p'(s) based on the score p(s) and the score q(s) according to the above formula (2), associates it with the word, and stores it in the score information 11c. This allows the word segmenter 11 to perform word segmentation based on the updated parameters (score information 11c) in preprocessing of the inference process of the document classifier 12.

[0087] An example of an algorithm for training the word segmenter 11 using the second method is shown in symbol H. Symbol H shows an example of an algorithm when the second method is applied to machine learning using the Viterbi algorithm. In symbol H, training data D of Require (precondition) is an example of training data including text 21a and class information 21b.

[0088] In the first line of code H, a vocabulary V is initialized with a set of strings that appear in D. The processing in lines 2 to 11 is repeated as long as the condition that the vocabulary size |V| is greater than K is satisfied. In line 3, the score p(s) corresponding to each token s in the vocabulary V is randomly initialized. The scores p(s) corresponding to all tokens s in the vocabulary V are combined to form a score θ V The processing in lines 4 to 10 is repeated while the variable m is between 1 and M.

[0089] The processing in lines 5 to 9 is an example of processing P21. In line 5, D is calculated by a method such as N-best division (for example, N=1) to find the optimal value of θ V The words are segmented using N (for example, 1) segmentation patterns using the scores of D' θVAt this time, θ V is an example of the score p(s) calculated in process P21.

[0090] In the sixth line, we use the D' obtained in the fifth line. θV and the class information 21b of the vocabulary V, the weight W of the vocabulary V is calculated. V is calculated. Weight W V is composed of scores q(s) corresponding to all tokens s in the vocabulary V. V may be expressed, for example, by the right side of the above formula (3).

[0091] In line 7, the score θ V and the weight W obtained in line 6 V The score θ is a weighted sum based on V ^ is calculated. At this time, θ V ^ is an example of the score p'(s) estimated in process P21.

[0092] In line 8, D is calculated by using the θ V The words are segmented using N (for example, 1) segmentation patterns using the score of ^, and D' (θV)^ is obtained.

[0093] In the 9th line, θ V is θ V The unigram probability counted by ^ is updated. Line 10 is the process of pruning the vocabulary V according to the estimated score p'(s). In line 10, for example, θ V The bottom |V|×R tokens of θ are removed from V. In line 11, V is re-estimated based on D and V in the same way as in lines 4 to 9.

[0094] As described above, the segmentation control unit 31 trains the word segmenter 11 by using a given text 21a and class information 21b and applying a score q(s) based on the frequency of occurrence of words in each class to the score information 11c, regardless of whether the word segmenter 11 has undergone machine learning.

[0095] The above-described processing by the segmentation control unit 31 can also be considered as part of the parameter update processing within the word segmenter 11 (machine learning model) that is executed during word segmentation by the word segmenter 11. Therefore, at least part of the functions of the segmentation control unit 31 may be included in the word segmenter 11.

[0096] Returning to the explanation of FIG. 5, the pseudo-class generating unit 33 generates class information 21b as a pre-processing before the segmentation control unit 31 trains the word segmenter 11.

[0097] For example, the pseudo class generation unit 33 generates a pseudo class for each of the texts 21a based on the texts 21a included in the training dataset 21. The pseudo class generation unit 33 stores the generated pseudo classes as class information 21b in the training dataset 21 in association with the texts 21a. Various known methods such as k-means clustering may be used to generate the pseudo classes.

[0098] According to the pseudo class generating unit 33, even if the provided training dataset 21 does not include the class information 21b, it is possible to generate a pseudo class for each of the texts 21a.

[0099] Therefore, the server 1 can apply the method according to one embodiment to an existing word segmenter by acquiring the text and the existing word segmenter, for example. For example, since it is possible to omit the process of preparing a new training dataset 21 and creating a new word segmenter 11, it is possible to reduce the processing time, processing resources used, and costs of the server 1.

[0100] The inference control unit 32 controls the execution of inference processing using the word segmenter 11 and the document classifier 12. For example, the inference control unit 32 inputs the inference dataset 22 into the word segmenter 11 to obtain a token set 23 including segmented text for each text. The inference dataset 22 may include multiple inference data, for example, multiple texts. The token set 23 may include, for each of the multiple texts, multiple words (tokens) that are the results of word segmentation based on the word information 11a and score information 11c.

[0101] The inference control unit 32 uses the multiple segmented texts included in the token set 23 as input data to execute inference processing by the document classifier 12, and obtains an inference result 24 that is a document classification result of the segmented texts (texts).

[0102] FIG. 11 is a diagram showing an example of a document classification result according to the training method of the word segmenter 110 or 11. FIG. 11 shows an example of a document classification result by the document classifier 12 executed using the word segmenter 110 shown in FIG. 1 (denoted as "FIG. 1"), the word segmenter 11 trained by the first method shown in FIG. 9 (denoted as "first method"), and the word segmenter 11 trained by the second method shown in FIG. 10 (denoted as "second method"). The document classifier 12 executes document classification for three languages, namely, Japanese, Chinese, and English, and the datasets used are Japanese datasets J-1 to J-4, Chinese datasets C-1 to C-3, and English datasets E-1 to E-3. Class information 21b is included in each dataset. Viterbi-Training and EM-Training are machine learning methods for word segmenters based on the Viterbi algorithm and the EM algorithm, respectively. The values ​​in the table shown in Figure 11 are the F1 value measurements (F1% ± SD) for the evaluation data in the document classification results. The values ​​in bold are results that exceed the values ​​in Figure 1.

[0103] Fig. 12 is a diagram showing an example of a document classification result according to the training method of the word segmenter 11 when pseudo labels are used. Fig. 12 shows an example of a document classification result executed using each of the "first method" and the "second method" when K-means clustering is used as the pseudo label creation method and the number of clusters is set to L=5, L=10, and L=15. Note that other conditions are the same as in Fig. 11.

[0104] 11 and 12, it can be seen that the first and second methods contribute to improving performance in document classification in all of the languages, Japanese, Chinese, and English, regardless of the machine learning method. As such, according to the method of one embodiment, it is possible to improve the versatility of the word segmenter 11 while maintaining performance above the trade-off level for the inference accuracy of the subsequent model.

[0105] As described above, according to the method of one embodiment, when training the word splitter 11 using the training dataset 21, the segmentation control unit 31 trains the word splitter 11 based on the bias in the occurrence frequency in multiple classes for each token contained in multiple segmented texts obtained by inputting text 21a into the word splitter 11 and segmenting text 21a using multiple different segmentation patterns corresponding to text 21a.

[0106] As a result, even in a usage scenario where the latter-stage model has not been determined, for example, it is possible to utilize the class information 21b used in the latter-stage model and train the word segmenter 11 based on the bias in the occurrence frequency of each token in multiple classes, thereby contributing to improving the performance of the latter-stage model. Therefore, it is possible to appropriately adjust the word segmenter 11 without relying on the latter-stage model.

[0107] Furthermore, for example, in the previously described method of simultaneously optimizing (adjusting) the word segmentation and the later-stage model by starting the word segmentation and the later-stage model in parallel, the word segmentation is optimized by performing two stages of processing: a stage of creating the word segmentation and a stage of simultaneously optimizing the word segmentation and the later-stage model. For example, when the previously described method is executed using a dataset of 50,000 sentences, the creation stage takes about one hour and the simultaneous optimization stage takes about six hours.

[0108] In contrast, in the first method, as training for the machine-learned (pre-created) word segmenter 11, a process of adjusting the score of the segmented text (each of a plurality of words) in the parameters of the word segmenter 11 is performed, thereby optimizing the word segmenter 11. In the first method, a score p'(s) is later assigned to the parameters of the created word segmenter 11. Therefore, in addition to the time required for creating the word segmenter 11, processing time is required for estimating and assigning the score p'(s), but this processing time is only about one minute.

[0109] In the second technique, in order to train the word segmenter 11, a process of adjusting the score of the segmented text (each of a plurality of words) in the parameters of the word segmenter 11 is performed at the creation stage of the word segmenter 11 (during the machine learning process), thereby optimizing the word segmenter 11. Since the second technique estimates and assigns the score p'(s) at the creation stage of the word segmenter 11, no processing time is required other than the time required for the creation stage of the word segmenter 11.

[0110] In this way, when either the first method or the second method is executed using a dataset of 50,000 sentences, the entire process of the simultaneous optimization stage can be omitted, which reduces the time by about 6 hours compared to the previously described methods. Therefore, according to the first and second methods, the optimization speed of the word segmenter 11 can be significantly improved compared to the previously described methods, and training of the word segmenter 11, which contributes to improving the performance of the subsequent model, can be achieved in a short time.

[0111] [D] Operation example Next, an example of the operation of the server 1 as an example of an embodiment configured as described above will be described with reference to FIGS.

[0112] [D-1] Training the word segmenter using the first method FIG. 13 is a flowchart for explaining an example of the operation of the training process for the word segmenter 11 using the first technique in the server 1 according to an embodiment.

[0113] 13, the segmentation control unit 31 calculates the probability p(s|c) for every possible token s based on the parameters (score information 11b) generated for the machine-learned word segmenter 11 and the training data included in the training dataset 21 (step S1). Note that if the training dataset 21 does not include class information 21b, the pseudo class generation unit 33 may generate a pseudo label as class information 21b based on the text 21a.

[0114] The division control unit 31 calculates the score q(s) based on the probability p(s|c) (step S2). Then, the division control unit 31 estimates the score p'(s) based on the score q(s), and updates the parameters by reflecting the score p'(s) in the parameter score information 11c (step S3), and the process ends.

[0115] [D-2] Training the word segmenter using the second method FIG. 14 is a flowchart for explaining an example of the operation of the training process for the word segmenter 11 using the second technique in the server 1 according to an embodiment.

[0116] As illustrated in Figure 14, the segmentation control unit 31 inputs the text 21a of the training data included in the training dataset 21 into the word segmenter 11, and estimates a score p(s) for each of the multiple words segmented by the word segmenter 11 using the frequency of occurrence of the word (step S11).

[0117] The division control unit 31 calculates the probability p(s|c) for every possible token s based on the estimated score p(s) and the training data included in the training dataset 21 (step S12). Note that if the training dataset 21 does not include class information 21b, the pseudo class generation unit 33 may generate a pseudo label as class information 21b based on the text 21a.

[0118] The division control unit 31 calculates the score q(s) based on the probability p(s|c) (step S13). Then, the division control unit 31 estimates the score p'(s) based on the score q(s), and updates the parameters by reflecting the score p'(s) in the parameter score information 11c (step S14), and the process ends.

[0119] [D-3] Inference processing FIG. 15 is a flowchart illustrating an example of the operation of the inference process by the server 1 according to an embodiment.

[0120] As illustrated in FIG. 15, the inference control unit 32 inputs the inference dataset 22 into the word segmenter 11 that has been trained using the first or second method, and obtains a token set 23 as segmented text (step S21).

[0121] The inference control unit 32 executes inference by the trained document classifier 12 using the token set 23 (step S22), obtains and outputs the inference result 24, and the process ends. The inference control unit 32 may evaluate the trained document classifier 12 based on the inference result 24. The evaluation result may be output to another computer, etc., together with the inference result 24.

[0122] [E] Other The technology according to the embodiment described above can be implemented with the following modifications and variations.

[0123] For example, the blocks 31 to 33 included in the server 1 shown in FIG. 5 may be merged in any combination, or may be divided into individual blocks.

[0124] 5 may be configured such that each processing function is realized by a plurality of devices cooperating with each other via a network. As an example, the word segmenter 11, document classifier 12, segmentation control unit 31, inference control unit 32, and pseudo-class generation unit 33 may be realized by an application server or a web server, and the memory area for storing word information 11a, score information 11b and 11c, and each data 21 to 24 may be realized by a database (DB) server. In this case, the web server, application server, and DB server may cooperate with each other via a network to realize the processing function of the server 1.

[0125] [F] Notes The following additional notes are provided regarding the above-described embodiments.

[0126] (Appendix 1) A lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, wherein the lexical segmenter outputs the segmented character string data including a plurality of character strings by segmenting input character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, training the lexical segmenter based on the bias in the frequency of occurrence of each character string in the plurality of classes for each character string included in a plurality of segmented character string data obtained by inputting the character string data to the lexical segmenter and segmenting the character string data using a plurality of different segmentation patterns corresponding to the character string data; An information processing program that causes a computer to execute processing.

[0127] (Appendix 2) The training process includes: and adjusting the scores of the plurality of character strings in the parameters of the lexical segmenter based on the bias of the occurrence frequency in the plurality of classes for each of the character strings included in the plurality of segmented character string data. An information processing program as described in Appendix 1.

[0128] (Appendix 3) The adjusting process includes: a process of adjusting the scores of the plurality of character strings included in the plurality of segmented character string data so that a character string having a greater bias in frequency of appearance in the plurality of classes has a higher score, the lexical segmenter outputs, for each of the plurality of input character string data, segmented character string data that maximizes the total score of the plurality of character strings based on the parameters; An information processing program as described in Appendix 2.

[0129] (Appendix 4) The adjusting process includes: performed on the parameters of the machine-learned lexical segmenter; An information processing program according to appendix 2 or appendix 3.

[0130] (Appendix 5) The adjusting process includes: When performing a machine learning process of the lexical segmenter, the process is performed on the parameters. An information processing program according to appendix 2 or appendix 3.

[0131] (Appendix 6) A lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, wherein the lexical segmenter outputs the segmented character string data including a plurality of character strings by segmenting input character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, training the lexical segmenter based on the bias in the frequency of occurrence of each character string in the plurality of classes for each character string included in a plurality of segmented character string data obtained by inputting the character string data to the lexical segmenter and segmenting the character string data using a plurality of different segmentation patterns corresponding to the character string data; An information processing method in which processing is performed by a computer.

[0132] (Appendix 7) The training process includes: and adjusting the scores of the plurality of character strings in the parameters of the lexical segmenter based on the bias of the occurrence frequency in the plurality of classes for each of the character strings included in the plurality of segmented character string data. 1. The information processing method described in Appendix 6.

[0133] (Appendix 8) The adjusting process includes: a process of adjusting the scores of the plurality of character strings included in the plurality of segmented character string data so that a character string having a greater bias in frequency of appearance in the plurality of classes has a higher score, the lexical segmenter outputs, for each of the plurality of input character string data, segmented character string data that maximizes the total score of the plurality of character strings based on the parameters; 10. The information processing method described in Appendix 7.

[0134] (Appendix 9) The adjusting process includes: performed on the parameters of the machine-learned lexical segmenter; 10. The information processing method according to claim 7 or 8.

[0135] (Appendix 10) The adjusting process includes: When performing a machine learning process of the lexical segmenter, the process is performed on the parameters. 10. The information processing method according to claim 7 or 8.

[0136] (Appendix 11) A lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, wherein the lexical segmenter outputs the segmented character string data including a plurality of character strings by segmenting input character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, training the lexical segmenter based on the bias in the frequency of occurrence of each character string in the plurality of classes for each character string included in a plurality of segmented character string data obtained by inputting the character string data to the lexical segmenter and segmenting the character string data using a plurality of different segmentation patterns corresponding to the character string data; An information processing device comprising a control unit.

[0137] (Appendix 12) The control unit, in the training process, adjusting the scores of the plurality of character strings in the parameters of the lexical segmenter based on the bias of the occurrence frequency in the plurality of classes for each of the character strings included in the plurality of segmented character string data; 12. The information processing device according to claim 11.

[0138] (Appendix 13) The control unit In the adjustment process, the scores of the plurality of character strings included in the plurality of segmented character string data are adjusted so that a character string having a greater bias in frequency of appearance in the plurality of classes has a higher score; the lexical segmenter outputs, for each of the plurality of input character string data, segmented character string data that maximizes the total score of the plurality of character strings based on the parameters; 13. The information processing device according to claim 12.

[0139] (Appendix 14) The control unit performing the adjustment process on the parameters of the machine-learned lexical segmenter; 14. The information processing device according to claim 12 or 13.

[0140] (Appendix 15) The control unit performing the adjusting process on the parameters when performing the machine learning process of the lexical segmenter; 14. The information processing device according to claim 12 or 13. [Explanation of symbols]

[0141] 1 server 10. Computers 11 Word Splitter 11a Word Information 11b, 11c score information 12 Document Classifier 21 Training Dataset 21a Text 21b Class Information 22 Inference Dataset 23 Token Set 24 Inference results 30 Control Unit 31 Division control unit 32 Inference control section 33 Pseudo-class generation part

Claims

1. A lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, wherein the lexical segmenter outputs the segmented character string data including a plurality of character strings by segmenting input character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, training the lexical segmenter based on the bias in the frequency of occurrence of each character string in the plurality of classes for each character string included in a plurality of segmented character string data obtained by inputting the character string data to the lexical segmenter and segmenting the character string data using a plurality of different segmentation patterns corresponding to the character string data; An information processing program that causes a computer to execute processing.

2. The training process includes: and adjusting the scores of the plurality of character strings in the parameters of the lexical segmenter based on the bias of the occurrence frequency in the plurality of classes for each of the character strings included in the plurality of segmented character string data. The information processing program according to claim 1 .

3. The adjusting process includes: a process of adjusting the scores of the plurality of character strings included in the plurality of segmented character string data so that a character string having a greater bias in frequency of appearance in the plurality of classes has a higher score, the lexical segmenter outputs, for each of the plurality of input character string data, segmented character string data that maximizes the total score of the plurality of character strings based on the parameters; The information processing program according to claim 2 .

4. The adjusting process includes: performed on the parameters of the machine-learned lexical segmenter; 4. The information processing program according to claim 2 or 3.

5. The adjusting process includes: When performing a machine learning process of the lexical segmenter, the process is performed on the parameters.

4. The information processing program according to claim 2 or 3.

6. A lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, wherein the lexical segmenter outputs the segmented character string data including a plurality of character strings by segmenting input character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, training the lexical segmenter based on the bias in the frequency of occurrence of each character string in the plurality of classes for each character string included in a plurality of segmented character string data obtained by inputting the character string data to the lexical segmenter and segmenting the character string data using a plurality of different segmentation patterns corresponding to the character string data; An information processing method in which processing is performed by a computer.

7. A lexical segmenter that outputs segmented character string data to be input to a machine learning model that performs inference processing, wherein the lexical segmenter outputs the segmented character string data including a plurality of character strings by segmenting input character string data, when the lexical segmenter is trained using a training dataset that includes a plurality of training data in which the character string data is associated with class information indicating to which of a plurality of classes the character string data belongs, training the lexical segmenter based on the bias in the frequency of occurrence of each character string in the plurality of classes for each character string included in a plurality of segmented character string data obtained by inputting the character string data to the lexical segmenter and segmenting the character string data using a plurality of different segmentation patterns corresponding to the character string data; An information processing device comprising a control unit.

Images