Embedding device, embedding method, and embedding program

Abstract

An embedding device according to the present invention extracts a chat patch from a chat history accumulated in a chat database. Furthermore, the embedding device divides the extracted patch into tokens and creates a fill-in-the-blank problem in which a portion of the tokens are masked. Subsequently, the embedding device trains a machine learning model that predicts (restores) the original values of the masked portion in the fill-in-the-blank problem. In addition, the embedding device outputs a vector representing user characteristics included in the chat history by using a user ID vectorization unit (13e) included in the machine learning model after training.

Description

Technical Field

[0001] The present invention relates to an embedding device, an embedding method, and an embedding program.

Background Art

[0002] In recent years, with the development of Internet communication, people's activities online through social media and the like have become active.

[0003] By using the online personal activities and communication history of an individual to digitize (vectorize) and characterize the individual, and by making use of it, it can be expected to be used in marketing suitable for the individual, matching with others, and the teaming strategy of an organization. Especially in matching and teaming, it is necessary to consider the characteristics of an individual's knowledge and communication in vectorizing the individual.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] However, although there are technologies for vectorizing an individual's knowledge characteristics and interest characteristics (see Non-Patent Document 1) and for vectorizing interaction characteristics (see Non-Patent Document 2), there is no established technology for constructing a vector that includes both of these. In other words, there is no established technology for constructing a vector that embeds both an individual's knowledge characteristics and interaction characteristics, which are necessary for matching, teaming, etc.

[0006] Therefore, an object of the present invention is to construct a vector that embeds both an individual's knowledge characteristics and interaction characteristics. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention provides a token vectorization unit that uses an extraction unit that extracts, for each chat channel, a patch that is part of a chat history including the channel ID of the chat, the user IDs of users who have made comments on the channel, and information indicating the content of comments in chronological order, a division unit that divides the patch into tokens of the minimum components, a masking unit that masks some of the tokens, a first vectorization unit that vectorizes user ID tokens, and a second vectorization unit that vectorizes other tokens, to vectorize each token of the patch including the masked tokens, and The system is characterized by comprising: a restoration unit that restores the original values ​​of the masked tokens from the results of vectorizing each token of the patch using a restoration model that restores the original values ​​of the masked tokens; a learning unit that trains the token vectorization unit and the restoration unit so that the values ​​of the tokens before masking are used as correct data and the original values ​​of the masked tokens can be restored with high accuracy by the token vectorization unit and the restoration unit; a vectorization unit that converts the user ID of the user to be vectorized into a vector using a first vectorization unit in the token vectorization unit after the learning; and an output processing unit that outputs the vector converted by the first vectorization unit. [Effects of the Invention]

[0008] According to the present invention, it is possible to construct a vector that embeds both the knowledge characteristics and the interaction characteristics of an individual. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining the flow of model learning by the embedding device. [Figure 2] FIG. 2 is a diagram for explaining the flow of vectorization of a user ID by the embedding device. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the embedding device. [Figure 4]FIG. 4 is a diagram showing an example of input data and output data of the patch extraction unit in FIG. [Figure 5] FIG. 5 is a diagram showing an example of input data and output data of the tokenization unit in FIG. [Figure 6] FIG. 6 is a diagram showing an example of input data and output data of the masking section in FIG. [Figure 7] FIG. 7 is a diagram illustrating an example of input data and output data of the token vectorization unit in FIG. [Figure 8] FIG. 8 is a diagram showing an example of input data and output data of the restoration unit in FIG. [Figure 9] FIG. 9 is a diagram showing an example of input data and output data to the user ID vectorization unit of FIG. 3 after learning. [Figure 10] FIG. 10 is a flowchart illustrating an example of a processing procedure for model learning by the embedding device. [Figure 11] FIG. 11 is a flowchart illustrating an example of a processing procedure for vectorizing a user ID by the embedding device. [Figure 12] FIG. 12 is a diagram illustrating an example of the configuration of a computer that executes an embedding program. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0011] [overview] The embedding device of this embodiment creates and outputs a vector in which both the user's personal knowledge and interest characteristics and the interaction characteristics with other users are quantified and embedded from the user's chat history.

[0012] The chat history includes, for example, for each chat channel, information such as the channel ID of the chat, the name of the channel, the user IDs of the users who made comments in the channel, and the content of the comments arranged in chronological order.

[0013] An overview of the embedding device will be explained using Figure 1. First, the embedding device extracts chat patches (chat patches) from a chat database that stores chat histories. A chat patch is, for example, a portion of comments within the same chat channel extracted from the chat history.

[0014] Next, the embedding device divides the extracted patch into tokens and masks some of the tokens (e.g., the user ID token).The embedding device then trains a machine learning model that infers (restores) the original values ​​of the masked tokens.In other words, the embedding device compares the mask restoration result output by the machine learning model with the value before masking (correct answer), and trains the machine learning model so that the mask restoration result approaches the correct answer.

[0015] The machine learning model includes, for example, a token vectorization unit 13d that vectorizes each token and a restoration unit 13h that restores the masked token value from the vector of each token. The token vectorization unit 13d includes a user ID vectorization unit 13e that outputs a vector of the feature amount of the user of the user ID based on the user ID, and a subword vectorization unit 13f that vectorizes information other than the user ID (for example, the content of chat messages, etc.).

[0016] In other words, the embedding device trains a machine learning model that restores the original values ​​of tokens masked in chat patches, thereby training the user ID vectorization unit 13e, which outputs a vector of the features of each user participating in the chat.

[0017] Now, let us move on to the explanation of Fig. 2. When the embedding device inputs the user ID of a user to the user ID vectorization unit 13e that has been learned as described above, a vector representing the user of the input user ID is output.

[0018] For example, when the embedding device trains a machine learning model including the user ID vectorization unit 13e, it uses, for example, a chat history including, for each chat channel, information such as the channel ID of the chat, the name of the channel, the user IDs of the users who made comments on the channel, and the content of the comments arranged in chronological order, and the following information about the user is embedded in the vector output by the user ID vectorization unit 13e after training:

[0019] What kind of other users is the user likely to come into contact with (what kind of interaction characteristics does the user have)? -What kind of comments is the user likely to make (what kind of knowledge and interest characteristics does he / she have)? In what environment (chat channel) is the user most likely to speak up?

[0020] In other words, the embedding device can output a vector in which both the knowledge characteristics and interaction characteristics of an individual user are embedded by learning the co-occurrence of elements that make up a chat patch.

[0021] [Configuration example] Next, an example of the configuration of the embedding device 10 will be described with reference to Fig. 3. The embedding device 10 includes an input / output unit 11, a storage unit 12, and a control unit 13.

[0022] The input / output unit 11 is an interface that controls the input and output of various information. The storage unit 12 stores data that the control unit 13 references when executing various processes. For example, the storage unit 12 includes a chat database that stores chat history. The chat database may be installed outside the embedded device 10.

[0023] The storage unit 12 also stores parameters of a model used when the token vectorization unit 13d performs vectorization, parameters of a model used when the restoration unit 13h restores the value of a token, and the like.

[0024] The control unit 13 controls the entire embedding device 10. The control unit 13 includes a patch extraction unit 13a, a token division unit 13b, a mask unit 13c, a token vectorization unit 13d, a restoration unit 13h, a learning unit 13i, a vectorization unit 13j, and an output processing unit 13k.

[0025] The patch extractor 13a extracts a part of the chat history from the chat database as a chat patch. As described above, a chat patch is, for example, a part of the comments made in the same chat channel extracted from the chat history.

[0026] For example, when the patch extraction unit 13a receives input of information from the chat database, it outputs information showing the chat channel ID, the channel name, the user IDs of the users who are commenting on the channel, and the content of the users' comments in the order in which they were commented, as shown in Figure 4.

[0027] Returning to the explanation of FIG. 3, the token division unit 13b divides the components of the chat patch extracted by the patch extraction unit 13a into tokens, which are the smallest units. For example, the token division unit 13b extracts and tokenizes the channel ID and user ID of the patch using regular expressions. The token division unit 13b also tokenizes other sentences of the patch using SentencePiece. The token division unit 13b divides the patch into tokens using, for example, the technology described in Reference 1 below.

[0028] Reference 1: Taku Kudo and John Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,” arXiv preprint arXiv:1808.06226 (2018).

[0029] For example, when the token division unit 13b receives input of a chat patch shown in "Input Example" in Fig. 5, it divides the components of the patch into the smallest unit of tokens, as shown in "Token Division Example," and inserts a separator [sep] after the channel ID, the channel name, the user ID, the user's comments, etc. Then, the token division unit 13b outputs information in which tokens including the separator [sep] are arranged in a row, as shown in "Output Example" in Fig. 5, for example.

[0030] Returning to the description of FIG. 3, the masking unit 13c masks some of the tokens in the chat patch. For example, the masking unit 13c replaces the user ID token among the tokens in the chat patch with a special token indicating that the token has been masked. Note that the masking unit 13c may mask randomly selected tokens among the tokens in the chat patch.

[0031] For example, when the masking unit 13c receives an input of a group of tokens shown in "Example Input" in FIG. 6, it replaces some of the tokens with a special token called [mask] and outputs the tokens.

[0032] Returning to the explanation of Fig. 3, the token vectorization unit 13d vectorizes each of the input tokens.

[0033] The token vectorization unit 13d vectorizes the tokens using an embedding layer, which is a basic neural network structure. The token vectorization unit 13d vectorizes the tokens using, for example, the technique described in Reference 2 below.

[0034] Reference 2: Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space,” arXiv preprint arXiv:1301.3781 (2013).

[0035] The token vectorization unit 13d includes, for example, a user ID vectorization unit (user ID vectorizer) 13e and a subword vectorization unit (subword vectorizer) 13f. The channel ID vectorization unit 13g shown by the dashed line may or may not be included in the token vectorization unit 13d, and the cases where it is included will be described later.

[0036] The user ID vectorization unit 13e converts the user ID token into a vector. For example, the user ID vectorization unit 13e converts the user ID token into a vector using a user ID vectorization model that receives the user ID token as input and outputs a vector of the user ID token.

[0037] The subword vectorization unit 13f converts, for example, subword tokens (e.g., channel name tokens, comment content tokens) into vectors. For example, the subword vectorization unit 13f converts subword tokens into vectors using a subword vectorization model that receives subword tokens as input and outputs vectors of the tokens.

[0038] The token vectorization unit 13d converts the input token into a vector using an appropriate vectorization unit (user ID vectorization unit 13e or subword vectorization unit 13f). For example, if the token is a user ID token, the token vectorization unit 13d converts the token into a vector using the user ID vectorization unit 13e, and if the token is a statement token, the token is converted into a vector using the subword vectorization unit 13f.

[0039] For example, the token vectorization unit 13d converts each of the tokens shown in the "input example" in FIG. 7 into a vector shown in the "output example" by an appropriate vectorization unit (user ID vectorization unit 13e or subword vectorization unit 13f).

[0040] Returning to the explanation of Fig. 3, the restoration unit (restorer) 13h restores the original token values ​​of the masked part of the patch. For example, the restoration unit 13h receives the vectors of each token of the patch output from the token vectorization unit 13d as input, estimates the values ​​of the masked tokens, and restores the values ​​of the masked tokens using the estimated values.

[0041] The restoration unit 13h infers the value of the masked token from the vector of each token shown in, for example, "input example" in Fig. 8. For example, the restoration unit 13h outputs the token shown in "output example" in Fig. 8.

[0042] For example, the restoration unit 13h takes the vector of each token as input, and uses a model (restoration model) that outputs a guessed value of the value of the masked token to guess the value of the masked token, and restores the value of the masked token using the guessed value.

[0043] The restoration unit 13h estimates the value of the masked token using, for example, the BERT model (see Reference 3), which is a neural network for natural language processing.

[0044] Reference 3: Devlin, Jacob, et al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 (2018).

[0045] Returning to the explanation of Fig. 3, the learning unit 13i uses the group of tokens in the patch before masking as correct answer data, and learns the token vectorization unit 13d and the restoration unit 13h (machine learning model) so that the token vectorization unit 13d and the restoration unit 13h can accurately estimate the original values ​​of the masked tokens.

[0046] For example, the learning unit 13i performs learning on the token vectorization unit 13d and the restoration unit 13h so that the estimated value of the masked token by the restoration unit 13h approaches the correct value (value before masking) as closely as possible.

[0047] For example, if the token masked by the masking unit 13c is a user ID token, the user ID vectorization model and subword vectorization model of the token vectorization unit 13d and the restoration model of the restoration unit 13h are trained so that the value of the user ID guessed by the restoration unit 13h is as close as possible to the correct user ID value.

[0048] The vectorization unit 13j obtains vectors representing user characteristics by the user ID vectorization unit 13e of the token vectorization unit 13d learned by the learning unit 13i. For example, the vectorization unit 13j selects a user ID to be vectorized from among user IDs included in the chat history used for learning. Then, the vectorization unit 13j inputs the selected user ID to the learned user ID vectorization unit 13e to obtain vectorized information of the user ID.

[0049] For example, when the learned user ID vectorization unit 13e receives input of a user ID (token indicating a user ID) shown in "Input Example" in Fig. 9, it outputs the vector shown in "Output Example." As a result, the vectorization unit 13j obtains a vector that expresses the characteristics of the user with the user ID obtained from the chat history.

[0050] Returning to the description of Fig. 3, the output processing unit 13k outputs the information vectorized by the vectorization unit 13j. For example, the output processing unit 13k outputs information in which the user ID is vectorized by the vectorization unit 13j.

[0051] According to such an embedding device 10, it is possible to output a vector in which both the knowledge characteristics and interaction characteristics of an individual user are embedded.

[0052] [Example of processing procedure] 10 and 11, an example of the processing procedure of the embedding device 10 will be described. First, the patch extractor 13a of the embedding device 10 extracts chat patches from the chat history in the chat database (S1 in FIG. 10: Extract chat patches).

[0053] Next, the token dividing unit 13b divides the components of the chat patch extracted in S1 into tokens, which are the smallest units (S2: Divide into tokens). After that, the masking unit 13c masks some of the tokens of the chat patch (S3: Mask some of the tokens).

[0054] Then, the learning unit 13i trains the token vectorization unit 13d and the restoration unit 13h so that the original values ​​of the tokens masked in S3 can be restored (S4).

[0055] Thereafter, the vectorization unit 13j receives input of the user ID of the user to be vectorized (S11 in FIG. 11). Note that the user ID input here is a user ID selected from the user IDs included in the chat history in the chat database.

[0056] Then, the vectorization unit 13j vectorizes the user ID received in S11 using the learned user ID vectorization unit 13e (S12). After that, the output processing unit 13k outputs information on the vectorized user ID (S13).

[0057] By performing the above processing, the embedding device 10 can output a vector in which both the knowledge characteristics and interaction characteristics of the individual user are embedded.

[0058] [Other embodiments] The token vectorization unit 13d of the embedding device 10 may further include, for example, a channel ID vectorization unit 13g (channel ID vectorizer) shown in Fig. 3. The channel ID vectorization unit 13g converts channel ID tokens into vectors. For example, the channel ID vectorization unit 13g converts channel ID tokens into vectors using a channel ID vectorization model that receives channel ID tokens as input and outputs vectors of tokens for the channel.

[0059] In this case, the learning unit 13i learns the token vectorization unit 13d including the channel ID vectorization unit 13g. Then, the vectorization unit 13j inputs the channel ID of the user to be vectorized to the channel ID vectorization unit 13g in the learned token vectorization unit 13d, thereby vectorizing the features of the chat channel. Note that in the embedding device 10, when learning of the channel ID vectorization unit 13g is mainly performed, the token to be masked by the masking unit 13c may be the channel ID.

[0060] The vector output by the learned channel ID vectorization unit 13g has the following information about the chat channel embedded therein:

[0061] -Information about the users who make up the channel. · Information about the main topic of the channel. The results of user interaction in that channel (e.g., the amount of conversation and which users in the channel are most active).

[0062] This allows the embedding device 10 to output a vector in which the features of the chat channel are embedded.

[0063] Although the embedding device 10 uses the chat history of each user to train the token vectorization unit 13d and the restoration unit 13h, the present invention is not limited to this. For example, the embedding device 10 may use each user's comments on social media or the like to train the token vectorization unit 13d and the restoration unit 13h.

[0064] [System configuration, etc.] Furthermore, the components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0065] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0066] [program] The embedding device 10 can be implemented by installing a program (embedding program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the embedding device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0067] 12 is a diagram showing an example of a computer that executes an embedded program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0068] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0069] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the embedding device 10 are implemented as program modules 1093 in which computer-executable codes are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the embedding device 10 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0070] Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as needed and executes them.

[0071] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network) or WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0072] 10. Implantation Devices 11 Input / output section 12 Storage section 13 Control Unit 13a Patch extraction section 13b Tokenization section 13c Mask section 13d Token Vectorization Unit 13e User ID vectorization unit (first vectorization unit) 13f Subword Vectorization Unit (Second Vectorization Unit) 13g Channel ID Vectorization Unit (Third Vectorization Unit) 13h Restore section 13i Learning Department 13j Vectorization Department 13k output processing section

Claims

1. an extracting unit that extracts, for each chat channel, a patch that is a part of the chat history including the channel ID of the chat, the user IDs of users who made comments on the channel, and information indicating the contents of the comments in chronological order; a splitter for splitting the patch into minimal constituent tokens; a masking unit that masks a part of the token; a token vectorization unit that vectorizes each of the tokens of the patch, including the masked token, using a first vectorization unit that vectorizes a token of a user ID and a second vectorization unit that vectorizes other tokens; a restoration unit that restores the original values ​​of the masked tokens from the vectorized results of each of the tokens of the patch using a restoration model that restores the original values ​​of the masked tokens from the vectorized results of each of the tokens of the patch; a learning unit that learns the token vectorization unit and the restoration unit so that the original values ​​of the masked tokens can be restored with high accuracy by using the values ​​of the tokens before masking as correct data; a vectorization unit that converts the user ID of the user who is the target of the vectorization into a vector using a first vectorization unit in the token vectorization unit after learning; an output processing unit that outputs the vector converted by the first vectorization unit; An embedding device comprising:

2. The mask portion is Among the tokens constituting the patch, the token of the user ID is masked.

2. The embedding device of claim 1.

3. The token vectorization unit further a third vectorization unit that converts the channel ID token into a vector; The vectorization unit further A third vectorization unit in the learned token vector unit converts the channel ID of the channel to be vectorized into a vector.

2. The embedding device of claim 1.

4. The mask portion is Among the tokens constituting the patch, the token of the channel ID is masked.

4. The embedding device of claim 3.

5. 1. An embedding method performed by an embedding device, comprising: extracting, for each chat channel, a patch that is a part of the chat history, including the channel ID of the chat, the user IDs of the users who made comments on the channel, and information indicating the contents of the comments in chronological order; dividing the patch into minimal constituent tokens; masking a portion of the token; a step of training the machine learning model, which includes a first vectorization model that converts user ID tokens into vectors, a second vectorization model that converts other tokens into vectors, and a restoration model that restores the original values ​​of the masked tokens from the results of vectorizing each of the tokens of the patch using the first vectorization model and the second vectorization model, so that the original values ​​of the masked tokens can be restored with high accuracy; converting the user ID of the user to be vectorized into a vector using the first vectorization model in the machine learning model after learning; outputting a vector transformed by the first vectorization model; An embedding method comprising:

6. An embedding program for causing a computer to function as the embedding device according to any one of claims 1 to 4.

Images