Character string divider, character string dividing system, vocabulary group generation method, and program

Abstract

To enable a new vocabulary group to be efficiently generated on the basis of an existing vocabulary group.SOLUTION: A character string divider comprises: an acquisition unit for acquiring a vocabulary group including a plurality of vocabularies associated with identifiers and a character string; character string dividing means for dividing the character string acquired by the acquisition unit into a plurality of vocabularies on the basis of the vocabulary group; an output unit for outputting a plurality of vocabularies obtained by the character string dividing means; and a generation unit for generating a new vocabulary group on the basis of an existing vocabulary group. Each of the plurality of vocabularies is associated with an embedded vector. For a vocabulary included in a common part with the existing vocabulary group among the plurality of vocabularies included in the new vocabulary group, the generation unit associates an embedded vector associated to a vocabulary included in the existing vocabulary group to the vocabulary of the new vocabulary group without altering it.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a string segmentation device, a string segmentation system, a vocabulary generation method, and a program. [Background technology]

[0002] Non-Patent Document 1 below discloses a technique for adding Japanese vocabulary to an existing model held by a tokenizer in order to make a model that does not support Japanese compatible with Japanese. Summary of the Invention [Problem to be solved by the invention]

[0003] However, the technology of Patent Document 1 adds Japanese vocabulary to an existing model, so the number of vocabulary words in the new model is the sum of the number of vocabulary words in the existing model and the number of Japanese vocabulary words, which causes a problem of increased memory costs due to an increase in the number of identifiers for LLMs (Large Language Models).Another possible method is to recreate a new model of Japanese vocabulary, but in this case, it is not possible to generate the new model efficiently.

[0004] In order to solve the above-mentioned problems of the conventional technology, an object of the present invention is to enable efficient generation of new vocabularies based on existing vocabularies. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, a string segmentation device according to one embodiment includes a vocabulary group having a plurality of vocabulary words associated with identifiers, an acquisition unit that acquires a string, a string segmentation means that, based on the vocabulary group, divides the string acquired by the acquisition unit into a plurality of vocabulary words, an output unit that outputs the plurality of vocabulary words acquired by the string segmentation means, and a generation unit that generates a new vocabulary group based on an existing vocabulary group, each of the plurality of vocabulary words being associated with an embedding vector, and for a vocabulary word included in the new vocabulary group that is included in a common portion with the existing vocabulary group, the generation unit associates the embedding vector associated with the vocabulary word included in the existing vocabulary group with the vocabulary word of the new vocabulary group without changing it. [Effects of the Invention]

[0006] According to an embodiment of the string segmentation device, new vocabulary sets can be efficiently generated based on existing vocabulary sets. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a string segmentation system according to an embodiment. [Figure 2] A diagram showing the hardware configuration of a server according to an embodiment. [Figure 3] Functional block diagram of a server according to an embodiment. [Figure 4] Functional block diagram of a server according to an embodiment. [Figure 5] A conceptual diagram of a string splitting process by a tokenizer included in a server according to an embodiment. [Figure 6] FIG. 10 is a conceptual diagram of a generation process performed by a generation unit included in a server according to an embodiment; [Figure 7] FIG. 10 is a diagram illustrating an example of a generation process performed by a generation unit included in a server according to an embodiment; [Figure 8] A flowchart showing an example of a procedure of a generation process by a generation unit included in a server according to an embodiment. [Figure 9] FIG. 1 is a diagram showing an example (first example) of a vocabulary set provided in a server according to an embodiment; [Figure 10] FIG. 2 is a diagram showing an example (second example) of a vocabulary set provided in a server according to an embodiment; [Figure 11] 1 is a flowchart illustrating an example of a procedure for vector setting processing by a generation unit included in a server according to an embodiment; [Figure 12] FIG. 10 is a diagram illustrating an example of a vector setting process performed by a generating unit included in a server according to an embodiment; [Figure 13] FIG. 1 is a diagram showing an example (first example) of a configuration of cooperation between a tokenizer and a machine learning model included in a server according to an embodiment. [Figure 14] FIG. 2 is a diagram showing an example (second example) of the configuration of cooperation between a tokenizer and a machine learning model included in a server according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] (An example of the system configuration of the character string segmentation system 10) FIG. 1 is a diagram showing an example of the system configuration of a character string segmentation system 10 according to an embodiment.

[0010] 1 includes a server 12 and a PC 14. The server 12 and the PC 14 are communicably connected to each other via a communication network 16. The communication network 16 may be, for example, a LAN, the Internet, or the like.

[0011] The PC 14 is an example of a terminal device. The PC 14 has an input device for inputting a character string. Examples of the input device include a keyboard and a touch panel. The PC 14 can transmit the character string input by the user using the input device to the server 12 via the communication network 16. The PC 14 may also have a voice recognition function for recognizing voice input by the user through a microphone, and in this case, the character string recognized by the voice recognition function may be transmitted to the server 12.

[0012] The server 12 is an example of a "string segmentation device." The server 12 receives a string transmitted from the PC 14 via the communication network 16 and segments the string into multiple vocabularies by a string segmentation process. The server 12 can then output the multiple vocabularies obtained by the string segmentation process to the PC 14 via the communication network 16.

[0013] The PC 14 can receive the multiple vocabularies transmitted from the server 12 (i.e., the multiple vocabularies obtained by the string division process) via the communication network 16 and perform various processes using the multiple vocabularies.

[0014] In the example shown in Figure 1, one terminal device (PC 14) is connected to the server 12 via the communication network 16, but in reality, the string splitting system 10 allows multiple terminal devices to connect to the server 12 via the communication network 16, and each of the multiple terminal devices can send a string to the server 12.

[0015] In addition, in the example shown in FIG. 1, PC 14 is used as an example of a terminal device for inputting a character string, but the terminal device is not limited to this and may be any other device (for example, a smartphone, a tablet terminal, an electronic whiteboard, an inkjet printer, an MFP, a video conferencing terminal, a projector, an omnidirectional imaging device, etc.) as long as it is capable of at least inputting and transmitting a character string.

[0016] (Hardware configuration of Server 12 and PC 14) 2 is a diagram showing the hardware configuration of the server 12 and the PC 14 according to one embodiment. Here, the hardware configuration common to the server 12 and the PC 14 will be described.

[0017] As shown in FIG. 2, the server 12 and the PC 14 are constructed by a computer and include a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0018] The CPU 501 controls the overall operation of the server 12 or the PC 14. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memories and printers. The network I / F 509 is an interface for data communication using the communication network 16. The data bus 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 2.

[0019] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0020] (Server 12 functional configuration) 3 and 4 are functional block diagrams of the server 12 according to one embodiment. Fig. 3 is a functional block diagram of the server 12 before the tokenizer 110-2 is generated. Fig. 4 is a functional block diagram of the server 12 after the tokenizer 110-2 is generated.

[0021] As shown in FIG. 3, the server 12 before the tokenizer 110-2 is generated includes an acquisition unit 101, an output unit 102, a generation unit 103, and a tokenizer 110-1.

[0022] The acquisition unit 101 acquires a character string transmitted from the PC 14 and received by the server 12 via the communication network 16 as a character string to be subjected to the character string division process.

[0023] The tokenizer 110-1 is an example of a "string splitting means." The tokenizer 110-1 has a vocabulary set 111-1. The vocabulary set 111-1 is an example of an "existing vocabulary set." The vocabulary set 111-1 has a plurality of vocabularies associated with identifiers. The vocabulary set 111-1 is realized, for example, by a JSON file. Based on the vocabulary set 111-1, the tokenizer 110-1 splits the string acquired by the acquisition unit 101 into a plurality of vocabularies (i.e., a plurality of morphemes) by performing a string splitting process (i.e., a tokenization process) using a well-known technique on the string acquired by the acquisition unit 101.

[0024] The output unit 102 outputs the plurality of vocabularies obtained by the tokenizer 110-1. The plurality of vocabularies output by the output unit 102 are transmitted by the server 12 to the PC 14 via the communication network 16.

[0025] The generation unit 103 can generate a new tokenizer 110. For example, the generation unit 103 can generate a tokenizer 110-2 as shown in Fig. 4 as a new tokenizer 110 for the server 12 having the tokenizer 110-1 as shown in Fig. 3. In this case, the generation unit 103 can generate a vocabulary set 111-2 that the tokenizer 110-2 has, based on the vocabulary set 111-1 that the tokenizer 110-1 has.

[0026] As shown in FIG. 4, after the tokenizer 110-2 is generated, the server 12 further includes the tokenizer 110-2 in addition to the components of the server 12 shown in FIG.

[0027] The tokenizer 110-2 is another example of a "string splitting means." The tokenizer 110-2 has a vocabulary group 111-2. The vocabulary group 111-2 is an example of a "new vocabulary group." The vocabulary group 111-2 has a plurality of vocabularies associated with identifiers. The vocabulary group 111-2 is realized, for example, by a JSON file. Based on the vocabulary group 111-2, the tokenizer 110-2 splits the string acquired by the acquisition unit 101 into a plurality of vocabularies (i.e., a plurality of morphemes) by performing a string splitting process using a well-known technique on the string acquired by the acquisition unit 101.

[0028] The server 12 shown in Fig. 4 includes a tokenizer 110-1 and a tokenizer 110-2. Therefore, the server 12 shown in Fig. 4 allows the user to select either the tokenizer 110-1 or the tokenizer 110-2 as the tokenizer to be used for the string splitting process.

[0029] Alternatively, the server 12 shown in FIG. 4 may automatically select either the tokenizer 110-1 or the tokenizer 110-2 as the tokenizer to be used for the string splitting process based on a predetermined judgment condition (for example, whether the string acquired by the acquisition unit 101 is in English or Japanese).

[0030] Alternatively, the server 12 shown in FIG. 4 may use the new tokenizer 110-2 as the tokenizer to be used for the character string splitting process, and may not use the existing tokenizer 110-1.

[0031] In the server 12 shown in FIG. 4, when the tokenizer 110-1 is selected, the character string acquired by the acquisition unit 101 is subjected to character string division processing by the tokenizer 110-1 based on the vocabulary group 111-1.

[0032] Furthermore, in the server 12 shown in FIG. 4, when the tokenizer 110-2 is selected, the character string acquired by the acquisition unit 101 is subjected to character string division processing by the tokenizer 110-2 based on the vocabulary group 111-2.

[0033] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above.

[0034] (Concept of string splitting by tokenizer 110) FIG. 5 is a conceptual diagram of a string division process performed by the tokenizer 110 included in the server 12 according to an embodiment.

[0035] As shown in FIG. 5, the tokenizer 110 includes a vocabulary set 111, an encoder 112, and a decoder 113.

[0036] The vocabulary group 111 is configured to have a plurality of vocabularies each associated with an identifier.

[0037] The encoder 112 performs a string division process on the input string based on the vocabulary group 111, thereby dividing the input string into a plurality of words and outputting the plurality of words.

[0038] For example, in the example shown in FIG. 5, when the character string "send" is input as input example 1, the encoder 112 splits the character string "send" into the vocabulary words "send" and "masu" based on the vocabulary group 111, and outputs an identifier string [50785, 7580] including identifiers of these multiple vocabulary words as output example 1.

[0039] The decoder 113 performs a string concatenation process on the input multiple vocabulary words based on the vocabulary group 111, thereby concatenating the input multiple vocabulary words to generate a string and outputting the string.

[0040] For example, in the example shown in FIG. 5, when an identifier string [40833, 7580] is input as input example 2, the decoder 113 generates a character string "return" by combining the vocabulary word "return" corresponding to the identifier

[40833] and the vocabulary word "masu" corresponding to the identifier

[7580] based on the vocabulary group 111, and outputs the character string "return" as output example 2.

[0041] (Concept of generation process by generation unit 103) FIG. 6 is a conceptual diagram of a generation process performed by the generation unit 103 included in the server 12 according to an embodiment.

[0042] As shown in FIG. 6, when generating the tokenizer 110-2, the generation unit 103 can generate a new vocabulary set 111-2 that the tokenizer 110-2 has, based on an existing vocabulary set 111-1 that the tokenizer 110-1 has.

[0043] Specifically, the generation unit 103 selects, from among the multiple vocabularies included in the new vocabulary set 111-2, Vocabulary included in the common part 111-2A with the existing vocabulary set 111-1 is copied to the new vocabulary set 111-2 without changing the identifiers.

[0044] Furthermore, the generation unit 103 assigns an arbitrary identifier to the vocabulary words included in the new vocabulary set 111-2 that are not included in the common part 111-2A with the existing vocabulary set 111-1 (i.e., the vocabulary words included in the non-common part 111-2B with the existing vocabulary set 111-1) and adds them to the new vocabulary set 111-2.

[0045] This allows the generation unit 103 to efficiently generate the new vocabulary set 111-2 based on the existing vocabulary set 111-1, without having to recreate the new vocabulary set 111-2.

[0046] Furthermore, when generating the new vocabulary set 111-2, the generation unit 103 does not include the uncommon part 111-1A of the existing vocabulary set 111-1 in the new vocabulary set 111-2, i.e., does not simply add the uncommon part 111-2B to the existing vocabulary set 111-1, thereby preventing the data size of the new vocabulary set 111-2 from becoming too large.

[0047] Furthermore, the generation unit 103 reuses the identifier used in the non-common part 111-1A of the existing vocabulary set 111-1 for the non-common part 111-2B of the new vocabulary set 111-2, thereby eliminating the need to assign a new identifier, and thus preventing the data size of the new vocabulary set 111-2 from becoming too large.

[0048] (An example of generation processing by the generation unit 103) FIG. 7 is a diagram showing an example of a generation process by the generation unit 103 included in the server 12 according to an embodiment.

[0049] 7, the generation unit 103 copies, from among the multiple vocabulary words included in the new vocabulary set 111-2, the vocabulary word "this" and the vocabulary word "important" included in the common part 111-2A with the existing vocabulary set 111-1, to the new vocabulary set 111-2 without changing the identifiers ("200" and "202"). This allows the generation unit 103 to efficiently generate the new vocabulary set 111-2 based on the existing vocabulary set 111-1 without recreating the new vocabulary set 111-2.

[0050] 7, for the vocabulary "tomorrow" and the vocabulary "hello", which are not included in the common part 111-2A with the existing vocabulary set 111-1, among the multiple vocabulary words included in the new vocabulary set 111-2, the generation unit 103 reuses the identifiers ("201" and "203") used in the non-common part 111-1A of the existing vocabulary set 111-1 (i.e., replaces the vocabulary of the identifiers) and adds them to the new vocabulary set 111-2. This eliminates the need for the generation unit 103 to assign new identifiers, thereby preventing the data size of the new vocabulary set 111-2 from becoming too large.

[0051] That is, the generation unit 103 can prevent the number of identifiers of the new vocabulary set 111-2 from increasing from the number of identifiers of the existing vocabulary set 111-1. This prevents the number of identifiers of the LLM corresponding to the new vocabulary set 111-2 from increasing from the number of identifiers of the LLM corresponding to the existing vocabulary set 111-1, thereby suppressing an increase in memory cost related to the LLM corresponding to the new vocabulary set 111-2.

[0052] (Example of procedure of generation process by generation unit 103) FIG. 8 is a flowchart showing an example of a procedure of a generation process by the generation unit 103 included in the server 12 according to an embodiment.

[0053] First, the generation unit 103 selects one vocabulary from a plurality of vocabularies included in the new vocabulary set 111-2 (step S801).

[0054] Next, the generation unit 103 determines whether the vocabulary selected in step S801 is included in the existing vocabulary set 111-1 (step S802).

[0055] If it is determined in step S802 that the vocabulary selected in step S801 is included in the existing vocabulary set 111-1 (step S802: YES), the generation unit 103 copies the vocabulary selected in step S801 to a new vocabulary set 111-2 without changing the identifier (step S803). After that, the generation unit 103 proceeds to step S806.

[0056] If it is determined in step S802 that the vocabulary selected in step S801 is not included in the existing vocabulary set 111-1 (step S802: NO), the generation unit 103 assigns the identifier used in the non-common part 111-1A of the existing vocabulary set 111-1 to the vocabulary selected in step S801 (step S804). Then, the generation unit 103 registers the vocabulary selected in step S801 and the identifier assigned in step S804 in the new vocabulary set 111-2 (step S805). Thereafter, the generation unit 103 proceeds to the process of step S806.

[0057] In step S806, the generation unit 103 determines whether or not all of the multiple vocabulary words included in the new vocabulary set 111-2 have been selected (step S806).

[0058] In step S806, if it is determined that all of the multiple vocabulary words included in the new vocabulary group 111-2 have not been selected (step S806: NO), the generation unit 103 returns the process to step S801.

[0059] If it is determined in step S806 that all of the multiple vocabulary words included in the new vocabulary group 111-2 have been selected (step S806: YES), the generation unit 103 ends the series of processes shown in FIG.

[0060] (Example of vocabulary group 111-1, 111-2 (Example 1)) FIG. 9 is a diagram showing an example (first example) of the vocabulary groups 111-1 and 111-2 provided in the server 12 according to an embodiment.

[0061] In the example shown in FIG. 9, an English vocabulary set including a plurality of English words (dog, cat, this, day, . . . ) is used as the existing vocabulary set 111-1.

[0062] In the example shown in FIG. 9, an English vocabulary group including a plurality of Japanese vocabulary words (dog, cat, this, day, . . . ) is used as the new vocabulary group 111-2.

[0063] That is, in the example shown in FIG. 9, the existing vocabulary set 111-1 and the new vocabulary set 111-2 are generated on a language-by-language basis.

[0064] (Example of vocabulary group 111-1, 111-2 (Example 2)) FIG. 10 is a diagram showing an example (second example) of the vocabulary groups 111-1 and 111-2 provided in the server 12 according to an embodiment.

[0065] In the example shown in FIG. 10, a vocabulary set of business terms including a plurality of business terminology terms (gross profit, business, KPI, . . . ) is used as the existing vocabulary set 111-1.

[0066] In the example shown in FIG. 10, a vocabulary group of everyday words including a plurality of everyday words (walk, bucket, blanket, . . . ) is used as the new vocabulary group 111-2.

[0067] That is, in the example shown in FIG. 10, the existing vocabulary set 111-1 and the new vocabulary set 111-2 are generated on a term-by-term basis.

[0068] (Example of procedure for vector setting process by generation unit 103) 11 is a flowchart showing an example of a procedure of a vector setting process by the generation unit 103 included in the server 12 according to an embodiment. For example, the generation unit 103 executes a series of processes shown in FIG. 11 after generating a new vocabulary group 111-2.

[0069] First, the generation unit 103 selects one vocabulary from a plurality of vocabularies included in the new vocabulary set 111-2 (step S1101).

[0070] Next, the generation unit 103 determines whether the vocabulary selected in step S1101 is included in the existing vocabulary set 111-1 (step S1102).

[0071] If it is determined in step S1102 that the vocabulary selected in step S1101 is included in the existing vocabulary set 111-1 (step S1102: YES), the generation unit 103 sets the embedding vector associated with the vocabulary in the existing LLM 120-1 to the identifier of the vocabulary selected in step S1101 in the new LLM 120-2 without changing it (step S1103).Then, the generation unit 103 proceeds to step S1106.

[0072] If it is determined in step S1102 that the vocabulary selected in step S1101 is not included in the existing vocabulary set 111-1 (step S1102: NO), the generation unit 103 divides the vocabulary selected in step S1101 into multiple vocabularies based on the existing vocabulary set 111-1 using the existing tokenizer 110-1 (step S1104). Then, the generation unit 103 sets the average value of multiple embedding vectors associated with the multiple vocabularies obtained in step S1104 in the existing LLM 120-1 to the identifiers of the vocabulary selected in step S1101 in the new LLM 120-2 (step S1105). Thereafter, the generation unit 103 proceeds to step S1106.

[0073] In step S1106, the generation unit 103 determines whether or not all of the multiple vocabulary words included in the new vocabulary group 111-2 have been selected (step S1106).

[0074] In step S1106, if it is determined that all of the multiple vocabulary words included in the new vocabulary group 111-2 have not been selected (step S1106: NO), the generation unit 103 returns the process to step S1101.

[0075] If it is determined in step S1106 that all of the multiple vocabulary words included in the new vocabulary group 111-2 have been selected (step S1106: YES), the generation unit 103 ends the series of processes shown in FIG.

[0076] (An example of vector setting processing by the generation unit 103) FIG. 12 is a diagram showing an example of vector setting processing by the generating unit 103 included in the server 12 according to an embodiment.

[0077] 12, an embedding vector is associated with each of the multiple vocabulary words in the existing vocabulary set 111-1. Specifically, the embedding vectors of the multiple vocabulary words in the existing vocabulary set 111-1 are provided in the existing LLM 120-1 corresponding to the existing vocabulary set 111-1, and are associated with the vocabulary words in the existing vocabulary set 111-1 by identifiers.

[0078] Similarly, an embedding vector is associated with each of the multiple vocabulary elements in the new vocabulary set 111-2. Specifically, the embedding vectors for the multiple vocabulary elements in the new vocabulary set 111-2 are provided in a new LLM 120-2 corresponding to the new vocabulary set 111-2, and are associated with the vocabulary elements in the existing vocabulary set 111-1 by identifiers.

[0079] In the example shown in FIG. 12, the generation unit 103 copies the vocabulary "mail" included in the existing vocabulary set 111-1, out of the multiple vocabulary words included in the new vocabulary set 111-2, to the new vocabulary set 111-2 without changing the identifier "2549." At this time, the generation unit 103 associates the embedding vector associated with the identifier "2549" of the vocabulary "mail" in the existing LLM 120-1 with the identifier "2549" of the vocabulary "mail" in the new LLM 120-2 without changing it. This allows the generation unit 103 to efficiently set embedding vectors of vocabulary words common to the existing vocabulary set 111-1 for the new LLM 120-2.

[0080] 12, the generation unit 103 reuses the identifier "13603" used for the vocabulary "XXX" in the uncommon part 111-1A of the existing vocabulary set 111-1 (i.e., replaces the vocabulary of the identifier) ​​to add the vocabulary "Queen" which is not included in the existing vocabulary set 111-1, among the multiple vocabulary words included in the new vocabulary set 111-2, to the new vocabulary set 111-2. At this time, the generation unit 103 splits the vocabulary "Queen" into the vocabulary "Woman" and the vocabulary "King" based on the existing vocabulary set 111-1, using the existing tokenizer 110-1. The generation unit 103 then associates the average value of the embedding vector associated with the identifier "30647" of the vocabulary word "woman" in the existing LLM 120-1 and the embedding vector associated with the identifier "30462" of the vocabulary word "king" in the existing LLM 120-1 with the identifier "13603" of the vocabulary word "queen" in the new LLM 120-2. This allows the generation unit 103 to efficiently generate and set, for the new LLM 120-2, embedding vectors of vocabulary words that are not common to the existing vocabulary set 111-1.

[0081] (Example of a configuration of cooperation between the tokenizer 110 and a machine learning model (first example)) 13 is a diagram showing an example (first example) of a configuration in which the tokenizer 110 provided in the server 12 according to an embodiment cooperates with a machine learning model. FIG. 13 shows an example of a configuration in which the tokenizer 110 cooperates with an LLM 120, which is an example of a machine learning model. The LLM 120 may be provided in the server 12, or may be provided in an external device to the server 12.

[0082] As shown in FIG. 13, the LLM 120 has an embedding layer 121, a hidden layer 122, and an output layer 123.

[0083] In the example shown in Figure 13, (1) a character string is input to the tokenizer 110. (2) The tokenizer 110 converts the input character string into an identifier string based on the vocabulary set 111 and outputs the identifier string to the LLM 120. (3) The LLM 120 generates an identifier string that follows the input identifier string and outputs the identifier string to the tokenizer 110. (4) The tokenizer 110 converts the identifier string input from the LLM 120 into a character string based on the vocabulary set 111 and outputs the character string.

[0084] (Example of a configuration of cooperation between the tokenizer 110 and a machine learning model (Example 2)) 14 is a diagram showing an example (second example) of the configuration of cooperation between the tokenizer 110 and a machine learning model provided in the server 12 according to an embodiment. FIG. 14 shows an example of the configuration when the tokenizer 110 cooperates with a Text-to-Image Model 130, which is another example of a machine learning model. Note that the Text-to-Image Model 130 may be provided in the server 12 or in an external device to the server 12.

[0085] As shown in FIG. 14, the Text-to-Image Model 130 includes an embedding layer 131, a noise generator 132, a noise removal model 133, and an output layer 134.

[0086] 14, (1) a character string is input to the tokenizer 110. (2) The tokenizer 110 converts the input character string into an identifier string based on the vocabulary set 111, and outputs the identifier string to the Text-to-Image Model 130. (3) The Text-to-Image Model 130 generates an image based on the input identifier string, and outputs the image.

[0087] In the examples shown in Figures 13 and 14, LLM 120 and Text-to-Image Model 130 are used as examples of machine learning models, but this is not limited to these, and image-to-text, music generation AI, etc. may also be used as machine learning models.

[0088] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0089] The devices described in the example are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, server 12 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communications link, including a network, shared memory, etc., and to perform the processes disclosed herein. [Explanation of symbols]

[0090] 10 String Splitting System 12 Servers 14 PC 16. Communication Networks 101 Acquisition Department 102 Output section 103 Generation part 110-1 Tokenizer 111-1 Vocabulary group 111-1A Uncommon parts 110-2 Tokenizer 111-2 Vocabulary group 111-2A Common parts 111-2B Uncommon parts 120, 120-1, 120-2 LLM 121 Embedding Layer 122 Hidden Layer 123 Output layer 130 Text-to-Image Model 131 Embedding Layer 132 Noise generation section 133 Noise Reduction Model 134 and the output layer [Prior art documents] [Non-patent literature]

[0091] [Non-Patent Document 1] Explanation of the Japanese LLM "ELYZA-japanese-Llama-2-7b" published by ELYZA: (1) Preparatory Study<https: / / zenn.dev / elyza / articles / 2fd451c944649d>

Claims

1. a vocabulary group having a plurality of vocabularies to which identifiers are associated; an acquisition unit that acquires a character string; a character string dividing means for dividing the character string acquired by the acquiring unit into a plurality of vocabularies based on the vocabulary group; an output unit that outputs the plurality of vocabularies obtained by the character string dividing means; a generation unit that generates a new vocabulary set based on the existing vocabulary set; Equipped with each of the plurality of vocabularies is associated with an embedding vector; The generation unit Among the multiple vocabularies included in the new vocabulary group, for vocabularies that are included in a common portion with the existing vocabulary group, the embedding vectors associated with the vocabulary included in the existing vocabulary group are associated with the vocabulary in the new vocabulary group without any changes. A character string dividing device characterized by:

2. The generation unit Among the multiple vocabularies included in the new vocabulary group, those vocabularies that are included in the common part with the existing vocabulary group are copied to the new vocabulary group without changing their identifiers, and those vocabularies that are not included in the common part with the existing vocabulary group are assigned arbitrary identifiers and added to the new vocabulary group.

2. The character string segmentation device according to claim 1.

3. The generation unit Among the multiple vocabulary words included in the new vocabulary group, those that are not included in the common part with the existing vocabulary group are assigned the identifiers of the vocabulary words included in the existing vocabulary group that are not included in the new vocabulary group, and are added to the new vocabulary group.

2. The character string segmentation device according to claim 1.

4. The generation unit Among the multiple vocabulary words included in the new vocabulary group, for vocabulary words that are not included in the common part with the existing vocabulary group, the vocabulary words are divided into multiple vocabulary words based on the existing vocabulary group, and the average value of the multiple embedding vectors associated with the multiple vocabulary words included in the existing vocabulary group is associated with the vocabulary words in the new vocabulary group.

2. The character string segmentation device according to claim 1.

5. The existing vocabulary group and the new vocabulary group are generated on a linguistic basis.

5. The character string dividing device according to claim 1.

6. The existing vocabulary group and the new vocabulary group are generated on a term-by-term basis.

5. The character string dividing device according to claim 1.

7. The identifiers of the plurality of vocabulary words obtained by the character string dividing means are output to a machine learning model, thereby obtaining a processing result for the identifiers of the plurality of vocabulary words by the machine learning model.

5. The character string dividing device according to claim 1.

8. a vocabulary group having a plurality of vocabularies to which identifiers are associated; an acquisition unit that acquires a character string; a character string dividing means for dividing the character string acquired by the acquiring unit into a plurality of vocabularies based on the vocabulary group; an output unit that outputs the plurality of vocabularies obtained by the character string dividing means; a generation unit that generates a new vocabulary set based on the existing vocabulary set; Equipped with each of the plurality of vocabularies is associated with an embedding vector; The generation unit Among the multiple vocabularies included in the new vocabulary group, for vocabularies that are included in a common portion with the existing vocabulary group, the embedding vectors associated with the vocabulary included in the existing vocabulary group are associated with the vocabulary in the new vocabulary group without any changes. A string splitting system comprising:

9. A vocabulary group generation method for generating a vocabulary group configured by associating an identifier with each of a plurality of vocabularies, the vocabulary group being used when a character string is divided into a plurality of vocabularies by a character string dividing means, comprising: a vocabulary group generating step of generating a new vocabulary group based on an existing vocabulary group; each of the plurality of vocabularies is associated with an embedding vector; In the vocabulary group generation step, Among the multiple vocabularies included in the new vocabulary group, for vocabularies that are included in a common portion with the existing vocabulary group, the embedding vectors associated with the vocabulary included in the existing vocabulary group are associated with the vocabulary in the new vocabulary group without any changes. A vocabulary generation method comprising:

10. A program for generating a vocabulary group configured by associating an identifier with each of a plurality of vocabularies, the vocabulary group being used when a character string is divided into a plurality of vocabularies by a character string dividing means, Computer, a generating unit that generates a new vocabulary group based on the existing vocabulary group; each of the plurality of vocabularies is associated with an embedding vector; The generation unit Among the multiple vocabularies included in the new vocabulary group, for vocabularies that are included in a common portion with the existing vocabulary group, the embedding vectors associated with the vocabulary included in the existing vocabulary group are associated with the vocabulary in the new vocabulary group without any changes. A program characterized by:

Images