Information processing device, information processing method, and information processing program

Abstract

An information processing device (100) infers an answer to a query. A generation unit (110) generates a word count model (41) and a third-order tensor count model (42) by counting words with parts of speech obtained from a pre-learning text (21). A correction unit (120) generates a word deletion count (31) and a third-order tensor deletion count (32) by counting words with parts of speech to be deleted that are obtained from a deletion text (22). The correction unit (120) subtracts the word deletion count (31) from the word count model (41), and subtracts the third-order tensor deletion count (32) from the third-order tensor count model (42).

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Large-scale language models (LLMs) are generated by pre-training them on large corpora. LLM is an abbreviation for Large Language Models. On the other hand, from the perspectives of privacy protection, copyright infringement, bias issues, or data updates, there is a desire to remove partial examples from a large corpus after a large language model has been generated (i.e., to pretend that it was not pre-trained), which is called unlearning. In conventional deep learning, if a subset of a large corpus is removed, the entire pre-training process must be redone. This results in a significant amount of time and enormous costs.

[0003] Patent Document 1 discloses a technique for dividing training data for a machine learning model into a first set containing private data and a second set not containing private data. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-193533 [Non-patent literature]

[0005] [Non-Patent Document 1] Okimura et al., "Anti-Learning at the Attribute Level in Natural Language Processing," Proceedings of the 30th Annual Meeting of the Association for Natural Language Processing, 2024. [Overview of the project] [Problems that the invention aims to solve]

[0006] Patent Document 1 describes training a machine learning model using a first set of data that includes private data, and then splitting that machine learning model into an input model and an output model. The output model is then trained using a second set of data. In this way, Patent Document 1 trains the output model by removing private information through so-called transfer learning. As a result, it takes a lot of time to train and there is a problem that unlearning cannot be completely reversed. In addition, various unlearning methods in natural language processing are described in Non-Patent Document 1, but it is not easy to completely and reversibly forget the data.

[0007] This disclosure aims to enable high-precision partial undoing of pre-trained models, in the sense of completely and reversibly forgetting them within a reasonable timeframe. [Means for solving the problem]

[0008] The information processing device related to this disclosure is A generation unit generates a word count model from the results of counting parts of speech words obtained from a pre-training text, which is a pre-training text, and generates a third-order tensor count model from the results of performing a third-order tensor count on the parts of speech words obtained from the pre-training text. The system includes a correction unit that generates a word deletion count by counting the words with parts of speech to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text, and generates a third-order tensor deletion count by performing a third-order tensor count on the words with parts of speech to be deleted, subtracts the word deletion count from the word count model, and subtracts the third-order tensor deletion count from the third-order tensor count model. [Effects of the Invention]

[0009] In the information processing apparatus according to the present disclosure, a count model consisting of counts is generated, and the counts generated from the deletion text are subtracted from the count model. Therefore, according to the information processing apparatus according to the present disclosure, it is possible to cancel partial learning from a pre-learned count model with high accuracy in a reasonable time, which has the effect of achieving this.

Brief Description of the Drawings

[0010] [Figure 1] The figure which shows the structural example of the information processing apparatus which concerns on Embodiment 1. [Figure 2] The flowchart which shows the example of the model generation correction process which concerns on Embodiment 1. [Figure 3] The figure which shows the third-order tensor model of the comparative example compared with the word count model and the third-order tensor count model which concern on Embodiment 1. <000009​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​[Modes for carrying out the invention]

[0011] The following description of this embodiment will be illustrated with reference to the figures. In each figure, identical or corresponding parts are denoted by the same reference numerals. In the description of the embodiment, the explanation of identical or corresponding parts will be omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or processing.

[0012] Embodiment 1. ***Explanation of the structure*** Figure 1 shows an example of the configuration of the information processing device 100 according to this embodiment. The information processing device 100 is a computer. The information processing device 100 includes a processor 910, as well as other hardware such as memory 921, auxiliary storage device 922, input interface 930, output interface 940, and communication device 950. The processor 910 is connected to the other hardware via signal lines and controls this other hardware.

[0013] The information processing device 100 comprises, as functional elements, a generation unit 110, a modification unit 120, an inference unit 130, and a storage unit 140. The storage unit 140 stores a word count model 41 and a third-order tensor count model 42. The word count model 41 and the third-order tensor count model 42 are sometimes collectively referred to as the count model 40.

[0014] The functions of the generation unit 110, the modification unit 120, and the inference unit 130 are implemented by software. The storage unit 140 is provided in the memory 921. The storage unit 140 may also be provided in the auxiliary storage device 922, or it may be distributed between the memory 921 and the auxiliary storage device 922.

[0015] The processor 910 is a device that executes an information processing program. The information processing program is a program that implements the functions of the generation unit 110, the modification unit 120, and the inference unit 130. The processor 910 is an integrated circuit (IC) that performs arithmetic processing. Specific examples of the processor 910 include CPUs, DSPs, and GPUs. IC stands for Integrated Circuit. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0016] Memory 921 is a storage device that temporarily stores data. Specific examples of memory 921 are SRAM or DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The auxiliary storage device 922 is a storage device for storing data. A specific example of the auxiliary storage device 922 is an HDD. Alternatively, the auxiliary storage device 922 may be a portable storage medium such as an SD® memory card, CF, NAND flash, flexible disk, optical disk, compact disk, Blu-ray® disc, or DVD. HDD is an abbreviation for Hard Disk Drive. SD® is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash®. DVD is an abbreviation for Digital Versatile Disk.

[0017] Input interface 930 is a port to which input devices such as a mouse, keyboard, or touch panel are connected. Specifically, input interface 930 is a USB terminal. Alternatively, input interface 930 may be a port connected to a LAN. USB stands for Universal Serial Bus, and LAN stands for Local Area Network.

[0018] Output interface 940 is a port to which the cable of an output device, such as a display, is connected. Specifically, output interface 940 is a USB terminal or an HDMI® terminal. Specifically, the display is an LCD. Output interface 940 is also called the display interface. HDMI® is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display.

[0019] The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, a telephone line, or Wi-Fi (registered trademark). Specifically, the communication device 950 is a communication chip or NIC. NIC is an abbreviation for Network Interface Card.

[0020] The information processing program is executed in the information processing device 100. The information processing program is loaded into the processor 910 and executed by the processor 910. Memory 921 stores not only the information processing program but also the OS. OS is an abbreviation for Operating System. The processor 910 executes the information processing program while executing the OS. The information processing program and the OS may also be stored in auxiliary storage device 922. The information processing program and OS stored in auxiliary storage device 922 are loaded into memory 921 and executed by the processor 910. Note that part or all of the information processing program may be incorporated into the OS.

[0021] The information processing device 100 may have multiple processors that replace the processor 910. These multiple processors share the task of executing the information processing program. Each processor is a device that executes the information processing program, just like the processor 910.

[0022] Data, information, signal values, and variable values ​​used, processed, or output by the information processing program are stored in memory 921, auxiliary storage device 922, or registers or cache memory within the processor 910.

[0023] The word "part" in "generation unit 110," "correction unit 120," and "inference unit 130" may be replaced with "circuit," "process," "procedure," "process," or "circuit." The information processing program causes the computer to execute the generation process, the correction process, and the inference process. The word "process" in "generation process," "correction process," and "inference process" may be replaced with "program," "program product," "computer-readable storage medium storing the program," or "computer-readable recording medium recording the program." Furthermore, the information processing method is performed by the information processing device 100 executing the information processing program. Information processing programs may be provided on a computer-readable recording medium. Alternatively, information processing programs may be provided as program products.

[0024] Figure 1 shows an example of the configuration of the information processing device 100, in which a model generation and modification function for generating and modifying models and an inference phase for performing inference using the models are performed in a single device. As another example of configuration, the model generation and modification function and the inference function may be mounted on separate devices. For example, the information processing device 100 may be a system comprising a first device and a second device. The information processing device 100 may also be an information processing system configuration in which the first device is equipped with a generation unit and a modification unit, and the second device is equipped with an inference unit.

[0025] ***Explanation of operation*** Next, the operation of the information processing device 100 according to this embodiment will be described. The operation procedure of the information processing device 100 corresponds to the information processing method. Furthermore, the program that realizes the information processing, which is the operation of the information processing device 100, corresponds to the information processing program. The information processing operation of the information processing device 100 includes a model generation and modification process that implements a model generation and modification function, and an inference process that implements an inference function. This embodiment describes the model generation and modification process, and the inference process corresponding to the model generation and modification process according to this embodiment will be described in Embodiment 2.

[0026] <Model generation and modification process> Figure 2 is a flowchart showing an example of the model generation and modification process according to this embodiment. The model generation and modification process comprises a generation process and a modification process.

[0027] <<Generation process: Steps S11 to S14>> In the generation process, the generation unit 110 generates a word count model 41 based on the result of counting the parts of speech words obtained from the pre-trained text 21. The generation unit 110 also generates a third-order tensor count model 42 based on the result of performing a third-order tensor count on the parts of speech words obtained from the pre-trained text 21. Specifically, it is as follows:

[0028] In step S11, the generation unit 110 obtains the pre-trained text 21, for example, via the input interface 930. The pre-trained text 21 is a text used for pre-training for model generation. The generation unit 110 performs morphological analysis on the pre-trained text 21 to obtain words with parts of speech. Words with parts of speech are sequences of multiple words, each with a part of speech.

[0029] In step S12, the generation unit 110 counts the words with parts of speech obtained from the pre-trained text 21. In step S13, the generation unit 110 performs a third-order tensor count on the words with parts of speech obtained from the pre-trained text 21. In step S14, the generation unit 110 generates a count model 40. Specifically, the generation unit 110 stores the count result from step S12 as a word count model 41 in the storage unit 140. The word count model 41 is frequency information for each word with its part of speech. Furthermore, the generation unit 110 stores the result of the third-order tensor count in step S13 as a third-order tensor count model 42 in the storage unit 140. The third-order tensor count model 42 is frequency information for each of three words, consisting of one predicate (what to do) and two arguments (subject, object, etc.).

[0030] <<Correction process: Steps S21 to S24>> In the correction process, the correction unit 120 generates a word deletion count 31 as the result of counting the words with parts of speech to be deleted obtained from the text to be deleted 22. The correction unit 120 also generates a third-order tensor deletion count 32 as the result of performing a third-order tensor count on the words with parts of speech to be deleted. Then, the correction unit 120 subtracts the word deletion count 31 from the word count model 41 and subtracts the third-order tensor deletion count 32 from the third-order tensor count model 42. Specifically, it is as follows:

[0031] In step S21, the modification unit 120 obtains the text to be deleted 22, for example, via the input interface 930. The text to be deleted 22 is the text to be deleted from the pre-trained text 21. The modification unit 120 performs morphological analysis on the text to be deleted 22 and obtains words with parts of speech to be deleted. As shown in Figure 2, the words to be deleted, each with a part of speech, are a sequence of multiple words, each with a corresponding part of speech.

[0032] In step S22, the modification unit 120 counts the words with parts of speech to be deleted and generates the result as the word deletion count 31. The word deletion count 31 is the frequency information for each of the words with parts of speech to be deleted.

[0033] In step S23, the modification unit 120 performs a third-order tensor count on the words with parts of speech to be deleted and generates the result as a third-order tensor deletion count 32. The third-order tensor deletion count 32 is frequency information for each of three words, consisting of one predicate (what to do) and two arguments (subject, object, etc.).

[0034] In step S24, the modification unit 120 subtracts the word deletion count 31 from the word count model 41 and subtracts the third-order tensor deletion count 32 from the third-order tensor count model 42. The modification unit 120 stores the word count model 41 and the third-order tensor count model 42 after the subtraction in the storage unit 140.

[0035] ***Description of the effects of this embodiment*** Figure 3 shows a comparative example of a third-order tensor model for comparison with the word count model 41 and the third-order tensor count model 42 according to this embodiment. The upper part of Figure 3 shows an example of generating a third-order tensor model for the comparative example. The third-order tensor model for the comparative example is generated from a word count, which is the result of counting words with parts of speech obtained from the pre-trained text 21, and a third-order tensor count, which is the result of performing a third-order tensor count on the words with parts of speech obtained from the pre-trained text 21. The third-order tensor model in the comparative example is specifically a PMI value calculated using word counts and third-order tensor counts. The PMI value, PMI(v,x,y), is calculated by (Equation 1) in the upper part of Figure 3. P represents the probability of each event. α is, for example, α=11. In this way, the PMI value (information quantity) is directly stored as the third-order tensor model in the comparative example. Therefore, if only some of the training data is deleted from the pre-trained text 21, the entire model needs to be retrained, i.e., the PMI value needs to be recalculated.

[0036] Here, PMI(v,x,y) is the self-mutual information quantity that measures the degree of relationship between events v,x, andy. PMI is an abbreviation for Pointwise Mutual Information. In PMI(v, x, y), v is a single predicate word (what to do), and x and y are two terms (such as the subject and the object).

[0037] In the lower part of FIG. 3, an example of decomposing the 3 - order tensor model of the comparative example into the word count model 41 and the 3 - order tensor count model 42 according to the present embodiment is shown. As shown in the lower part of FIG. 3, the 3 - order tensor model PMI(w i , w j , Y k ) of the comparative example can be represented by the word count model C(w i ), C(w j ), C(Y k ) and the 3 - order tensor count model C(w i , w j , Y k ) as shown in (Equation 2). C indicates the count number of each event. N is the total frequency of all words, and α = 11.

[0038] In the information processing apparatus 100 according to the present embodiment, during learning, the word count model 41 and the 3 - order tensor count model 42 are generated as the count model 40 without calculating the PMI value. Although it will be described in detail in Embodiment 2, by generating the count model 40, at the time of inference, the PMI value is calculated using the word count model 41 and the 3 - order tensor count model 42, and inference is performed.

[0039] In the information processing apparatus 100 according to the present embodiment, a word deletion count and a 3 - order tensor deletion count are generated for the deletion text, and can be subtracted from each of the word count model 41 and the 3 - order tensor count model 42. That is, in the information processing apparatus 100 according to the present embodiment, the input deletion text is decomposed into count numbers, and by subtracting the count numbers, it is possible to return to the state where only the deletion text is not learned. Therefore, according to the information processing apparatus 100 according to the present embodiment, the deletion text can be canceled from the model obtained by learning the pre - learning text with high accuracy in a short time. Furthermore, P(v), P(x), and P(y), which are necessary for PMI calculation, all change when N is subtracted. Therefore, in a model that has P(v), P(x), and P(y), a complete recalculation is required when N is subtracted. On the other hand, if the model is stored with C(v), C(x), and C(y), recalculation during model training becomes unnecessary. In the information processing device 100 according to this embodiment, partial undoing is possible in a reasonable amount of time by storing the sum as N in the word count model.

[0040] ***Other configurations*** In this embodiment, the functions of the generation unit 110, the modification unit 120, and the inference unit 130 are implemented in software. As a modified example, the functions of the generation unit 110, the modification unit 120, and the inference unit 130 may be implemented in hardware. Specifically, the information processing device 100 includes an electronic circuit 909 in place of the processor 910.

[0041] Figure 4 shows an example of the configuration of an information processing device 100 according to a modified example of this embodiment. The electronic circuit 909 is a dedicated electronic circuit that implements the functions of the generation unit 110, the modification unit 120, and the inference unit 130. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0042] The functions of the generation unit 110, the modification unit 120, and the inference unit 130 may be implemented in a single electronic circuit, or they may be implemented in a distributed manner across multiple electronic circuits.

[0043] As another variation, some functions of the generation unit 110, the modification unit 120, and the inference unit 130 may be implemented by electronic circuits, while the remaining functions are implemented by software. Alternatively, some or all functions of the generation unit 110, the modification unit 120, and the inference unit 130 may be implemented by firmware.

[0044] Each of the processor and electronic circuit is also called a processing circuit. In other words, the functions of the generation unit 110, the modification unit 120, and the inference unit 130 are realized by the processing circuit.

[0045] Embodiment 2. This embodiment will primarily describe the points added to Embodiment 1. In this embodiment, components having the same function as those in Embodiment 1 are denoted by the same reference numerals, and their descriptions are omitted.

[0046] ***Explanation of the structure*** The configuration of the information processing device 100 according to this embodiment is the same as in Embodiment 1.

[0047] ***Explanation of operation*** This embodiment will describe the inference process.

[0048] <Inference Processing> Figure 5 is a flowchart showing an example of the inference process according to this embodiment. In the inference process, the inference unit 130 obtains a question text 23, which is text representing the question. Based on the part-of-speech words for the question obtained from the question text 23, the word count model 41, and the third-order tensor count model 42, the inference unit 130 infers the answer to the question. Specifically, it is as follows:

[0049] In step S31, the inference unit 130 obtains the question text 23, for example, via the input interface 930. The question text 23 is, for example, text representing a question. In the example in Figure 3, the question text 23 is shown as "Where does Taro XX live?". The inference unit 130 performs morphological analysis on the question text 23 and obtains words with parts of speech for the question. As shown in Figure 5, the words with parts of speech for the question are sequences of multiple words, each with a part of speech assigned to it.

[0050] In step S32, the inference unit 130 generates a word question count 33 as the result of counting the part-of-speech words for the question. In step S33, the inference unit 130 generates a second-order tensor count 34 as the result of performing a second-order tensor count on the part-of-speech words used for the question. Specifically, the second-order tensor count 34 is frequency information for each of two words, consisting of one predicate word (what to do) and one argument word.

[0051] In step S34, the inference unit 130 infers the answer to the question by calculating the PMI value based on the word question count 33, the second-order tensor count 34, the word count model 41, and the third-order tensor count model 42. Alternatively, the inference unit 130 may infer the answer to the question by calculating the PMI value based on the second-order tensor count 34, the word count model 41, and the third-order tensor count model 42. In step S35, the inference unit 130 outputs the sequence of candidate answer words obtained from the inference as the inference result 35. The sequence of candidate answer words may also be output ranked by likelihood, which indicates plausibility.

[0052] Specifically, the inference unit 130 calculates the PMI value of the word candidates using the following procedure. (1) Generate all words from the word count model 41. (2) Generate a search key for a third-order tensor from (1) and the second-order tensor count of 34. (3)(2) The count is retrieved from the third-order tensor count model 42 using the search key. (4) Using the total number of words, word frequency, and (3), calculate the PMI value of the answer word candidates.

[0053] Figure 6 shows a specific example of the inference process according to this embodiment. PMI(w i ,w j ,Y k ) in Y k The symbols indicate possible answer words. As explained in (Equation 2) of Embodiment 1, PMI(w i ,w j ,Y k This can be represented by the counts of the word count model 41 and the third-order tensor count model 42. The inference unit 130 takes the PMI value of the answer word candidate as input and calculates Y k Likelihood L(Y k Calculate Y. k Likelihood L(Y k ) is expressed by (Equation 3). Then, the inference unit 130 uses (Equation 4) to determine Y k Likelihood L(Y k The inference unit 130 outputs answer word candidates where Y is above the threshold. k Likelihood L(Y k The system may output candidate answer words whose likelihood is above the threshold, sorted by likelihood. If the likelihood of all candidates is below the threshold, the inference unit 130 may output a message such as "No information available."

[0054] In the example in Figure 6, for the question text "What animals can be kept as pets at home?", the inference result 35 outputs candidate answer words Y1 to Y5. Alternatively, the candidate answer words Y1 to Y5 could be ranked and output by likelihood.

[0055] ***Description of the effects of this embodiment*** In the information processing device 100 according to this embodiment, during training, the PMI value is not calculated, and a word count model and a third-order tensor count model are generated as count models. During inference, the PMI value of the answer word candidate is calculated using the word count model and the third-order tensor count model, and the answer word of the question is inferred. Therefore, the information processing device 100 according to this embodiment has the effect of being able to infer the answer to a question using a count model that undoes deleted text with high accuracy in a short amount of time.

[0056] Embodiment 3. This embodiment will primarily describe the differences from Embodiments 1 and 2, as well as the additions made to Embodiments 1 and 2. In this embodiment, components having the same function as those in Embodiments 1 and 2 are denoted by the same reference numerals, and their descriptions are omitted.

[0057] ***Explanation of the structure*** Figure 7 shows an example of the configuration of the information processing device 100 according to this embodiment. In addition to the configuration described in Embodiment 1, the information processing device 100 according to this embodiment stores the ngram count model 43 in the storage unit 140.

[0058] ***Explanation of operation*** Figure 8 is a flowchart showing an example of the model generation and modification process according to this embodiment. Figure 8 illustrates the generation and modification processes for the ngram count model 43, which is newly added to the model generation and modification process described in Embodiment 1. The generation and modification processes for the word count model 41 shown in Figure 8 are the same as in Embodiment 1. In this embodiment as well, the third-order tensor count model 42 is generated and modified, but since it is the same as in Embodiment 1, the explanation is omitted.

[0059] <Model generation and modification process> <<Generation Process>> In step S43, the generation unit 110 performs n-gram counting on the parts-of-speech words obtained from the pre-trained text 21. An n-gram is a method of dividing a part of a sentence into sequences of any number of words n. Here, frequency information of n-grams, which are sequences of any number of words n on the parts-of-speech words obtained from the pre-trained text 21, is obtained. In step S44, the generation unit 110 generates an ngram count model 43 as the result of counting ngrams for parts of speech words obtained from the pre-trained text 21, and stores it in the memory unit 140. The ngram count model 43 is frequency information of ngrams for parts of speech words obtained from the pre-trained text 21.

[0060] <<Correction Process>> In step S53, the modification unit 120 generates an ngram deletion count 36 as the result of counting ngrams for the part-of-speech words to be deleted. The ngram deletion count 36 is frequency information of ngrams for the part-of-speech words to be deleted. In step S54, the modification unit 120 subtracts the ngram deletion count 36 from the ngram count model 43.

[0061] Figure 9 is a flowchart showing an example of the inference process according to this embodiment. Figure 9 illustrates the inference process using the ngram count model 43, which is newly added to the inference process described in Embodiment 1.

[0062] <Inference Processing> In step S63, the inference unit 130 performs a second-order tensor count on the part-of-speech words used in the question and generates a second-order tensor count 34. The inference unit 130 also performs an n-gram count on the part-of-speech words used in the question and generates an n-gram question count 39. The n-gram question count 39 is the frequency information of n-grams for the part-of-speech words used in the question.

[0063] In step S64, the inference unit 130 infers the answer to the question based on the word question count 33, the second-order tensor count 34, the n-gram question count 39, the n-gram count model 43, the word count model 41, and the third-order tensor count model 42. In step S65, the inference unit 130 outputs the sequence of candidate answer words obtained as a result of the inference as the inference result 35.

[0064] Specifically, the inference unit 130 calculates the PMI value of the word candidates using the following procedure. (1) Generate all words from the word count model 41. (2) Generate a search key for a third-order tensor from (1) and the second-order tensor count of 34. (3) The number of third-order tensors is retrieved from the third-order tensor count model 42 using the search key from (2). (4) Using the total number of words, word frequency, and (3), calculate the PMI value of the answer word candidates. (5) The likelihood is calculated by a weighted sum of the ngrams using the ngram question count 39 and the ngram count model 43.

[0065] Figure 10 shows a specific example of the inference process according to this embodiment. The thick square box in Figure 10 shows the probability P of an n-gram. The dotted square box in Figure 10 shows the formula for expressing the probability P of an n-gram in terms of the n-gram count C. In this way, n-grams can also be broken down into counts and stored.

[0066] By decomposing the probabilities of n-grams into counts and storing them, a language model is constructed, enabling the estimation of word candidates for any given position, while also allowing for unlearning. Figure 10 shows the following process. (a) A forward-facing ngram with W n We estimate the candidates. In this process, we replace the n-gram probability with the counts of n-grams and n-1-grams during training. (b) W in backward ngram n We estimate the candidates. In this process, we replace the n-gram probability with the counts of n-grams and n-1-grams during training. (c) Equation 5 is a weighted sum of the predicate terms PMI for the AND candidates of (a) and (b) W n This is the formula for estimating the value. Here, β is the weighting coefficient.

[0067] ***Description of the effects of this embodiment*** As described above, the information processing device 100 according to this embodiment allows the use of an ngram count model as a count model, in addition to the functions of embodiments 1 and 2. Furthermore, the information processing device 100 according to this embodiment can calculate word counts and ngram counts from deleted text and modify the word count model and ngram count model. In this way, the ngram probability values ​​are not directly stored in the model database, but are instead decomposed and stored as an ngram count model, consisting of ngrams (2grams) and word counts (unigram counts).

[0068] Therefore, according to the information processing device 100 of this embodiment, deleted text can be undone with high accuracy in a short amount of time using the ngram count model. Furthermore, because the information processing device 100 of this embodiment can use the ngram count model, it has the effect of enabling more accurate inference.

[0069] Embodiment 4. This embodiment will primarily describe additions to those described in Embodiments 1 to 3. In this embodiment, components having the same function as those in Embodiments 1 to 3 are denoted by the same reference numerals, and their descriptions are omitted.

[0070] ***Explanation of the structure*** The configuration of the information processing device 100 according to this embodiment is the same as in Embodiment 1. Embodiment 1 described a method for removing portions from the pre-trained text that you want to delete from the model. This embodiment describes a method for adding portions to the pre-trained text that you want to add to the count model.

[0071] ***Explanation of operation*** Figure 11 is a flowchart showing an example of the model generation and modification process according to this embodiment. In this embodiment, the text to be deleted 22, as described in Embodiment 1, is replaced with the text to be added 24 to be added to the pre-trained text 21. Other aspects are the same as in Embodiment 1. The model generation process within the model generation and modification process is the same as in Embodiment 1. The inference process is also the same as in Embodiment 1. In the modification process of the model generation modification process, the modification unit 120 obtains the additional text 24 and performs the same processing as in Embodiment 1.

[0072] In step S21, the modification unit 120 obtains additional text 24, which is text to be added to the pre-learned text 21. The modification unit 120 obtains additional words with parts of speech by performing morphological analysis on the additional text 24. In step S22, the modification unit 120 generates the result of counting the words with parts of speech to be added as the word addition count 37. In step S23, the modification unit 120 generates a third-order tensor addition count 38 as the result of performing a third-order tensor count on the additional part-of-speech words. In step S24, the modification unit 120 adds a word addition count 37 to the word count model 41 and adds a third-order tensor addition count 38 to the third-order tensor count model 42.

[0073] ***Description of the effects of this embodiment*** The information processing device 100 according to this embodiment has the effect of being able to add events that you want to add to the pre-trained data to the count model with high accuracy in a reasonable amount of time.

[0074] Embodiment 5. This embodiment will primarily describe the points added to Embodiments 1 to 4. In this embodiment, components having the same function as those in Embodiments 1 to 4 are denoted by the same reference numerals, and their descriptions are omitted.

[0075] ***Explanation of the structure*** The configuration of the information processing device 100 according to this embodiment is the same as in Embodiment 1. In this embodiment, the pre-trained text 21 is tagged text. The deleted text 22 and added text 24 are extracted from the tagged text.

[0076] Figure 12 shows an example of tagged text according to this embodiment. The tagged text shown in Figure 12 is from the C4(ja) corpus. The text is tagged with URL or timestamp information. By using such tagged text, it becomes possible to specify, for example, the time period to be deleted or added, and to exclude old text.

[0077] ***Explanation of operation*** Figure 13 is a flowchart showing an example of the model generation and modification process according to this embodiment. The model generation and modification process and the inference process are the same as in Embodiment 1. In step S01, the pre-trained text 21 is extracted from the pre-trained tagged text. Furthermore, in step S02, the text to be deleted 22 and the text to be added 24 are extracted from pre-trained tagged text or other tagged text.

[0078] In Figure 13, the text to be deleted 22 and the text to be added 24 are input for the modification process, but the modification process for each of the texts, deleted 22 and added 24, is the same as in Embodiments 1 and 4.

[0079] The generation unit 110 generates a tagged word count model 41 and a third-order tensor count model 42 from the tagged pre-trained text 21. The modification unit 120 generates a tagged word deletion count 31 and a third-order tensor deletion count 32 from the tagged deletion text 22. The modification unit 120 subtracts the word deletion count 31 from the word count model 41 using the tags. The modification unit 120 also subtracts the third-order tensor deletion count 32 from the third-order tensor count model 42 using the tags. The modification unit 120 generates tagged word addition counts 37 and third-order tensor addition counts 38 from the tagged additional text 24. The modification unit 120 adds the word addition counts 37 to the word count model 41 using the tags. The modification unit 120 also adds the third-order tensor addition counts 38 to the third-order tensor count model 42 using the tags.

[0080] ***Description of the effects of this embodiment*** The information processing device 100 according to this embodiment has the effect of allowing the model's learning content to be freely edited using tags.

[0081] Embodiment 6. This embodiment will primarily describe the points added to Embodiments 1 to 5. In this embodiment, components having the same function as those in Embodiments 1 to 5 are denoted by the same reference numerals, and their descriptions are omitted.

[0082] ***Explanation of the structure*** The configuration of the information processing device 100 according to this embodiment is the same as in Embodiment 1. In this embodiment, similar to Embodiment 5, the pre-learned text 21 is tagged text, and the deleted text 22 and added text 24 are extracted from the tagged text. The tags include the domain type. For example, the text may be categorized into fields such as art, science, or politics.

[0083] In this embodiment, a pre-trained text 21 is used to generate a word count model 41 and a third-order tensor count model 42 for each domain.

[0084] ***Explanation of operation*** Figure 14 is a flowchart showing an example of the inference process according to this embodiment. In the model generation and modification process, the generation process processes the pre-trained text 21 for each domain to generate a word count model 41 and a third-order tensor count model 42 for each domain. In the modification process of the model generation modification process, the deleted text 22 or added text 24 is processed for each domain, and subtraction or addition is performed on the word count model 41 and the third-order tensor count model 42 for each domain. In Figure 14, a word count model 41 and a third-order tensor count model 42 are generated for each of domains 1, 2, and 3.

[0085] <Inference Processing> In step S74, the inference unit 130 obtains domain-specific weights for the word count model 41 and the third-order tensor count model 42. The inference unit 130 uses the word count model and the third-order tensor count model according to the domain weights to infer the answer to the question. The inference unit 130 may obtain the weights via the input interface 930. Alternatively, it may have a function that allows selecting weights according to predetermined conditions.

[0086] ***Description of the effects of this embodiment*** According to the information processing device 100 of this embodiment, it has the effect of switching between the learning content of models in multiple fields and changing their weighting.

[0087] In embodiments 1 to 6 described above, each part of the information processing device was described as an independent functional block. However, the configuration of the information processing device does not have to be as described in the embodiments above. The functional blocks of the information processing device can be configured in any way as long as they can realize the functions described in the embodiments above. As mentioned above, the information processing device does not have to be a single device, but may be a system composed of multiple devices. Furthermore, multiple parts of Embodiments 1 to 6 may be combined and implemented. Alternatively, only one part of these embodiments may be implemented. In addition, these embodiments may be combined and implemented in any way, either as a whole or in part. In other words, in embodiments 1 to 6, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment.

[0088] The embodiments described above are essentially preferred examples and are not intended to limit the scope of the Disclosure, the scope of the Applications of the Disclosure, or the scope of Uses of the Disclosure. The embodiments described above can be modified in various ways as needed. For example, the procedures described using flowcharts or sequence diagrams may be modified as appropriate. [Explanation of Symbols]

[0089] 21 Pre-trained text, 22 Deleted text, 23 Question text, 24 Added text, 31 Word deletion count, 32 Third-order tensor deletion count, 33 Word question count, 34 Second-order tensor count, 35 Inference result, 36 N-gram deletion count, 37 Word addition count, 38 Third-order tensor addition count, 39 N-gram question count, 40 Count model, 41 Word count model, 42 Third-order tensor count model, 43 N-gram count model, 100 Information processing device, 110 Generation unit, 120 Modification unit, 130 Inference unit, 140 Memory unit, 909 Electronic circuit, 910 Processor, 921 Memory, 922 Auxiliary storage device, 930 Input interface, 940 Output interface, 950 Communication device.

Claims

1. A generation unit generates a word count model from the results of counting parts of speech words obtained from a pre-training text, which is a pre-training text, and generates a third-order tensor count model from the results of performing a third-order tensor count on the parts of speech words obtained from the pre-training text. The modification unit generates a word deletion count by counting the words with parts of speech to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text, and generates a third-order tensor deletion count by performing a third-order tensor count on the aforementioned words with parts of speech to be deleted, subtracts the word deletion count from the word count model, and subtracts the third-order tensor deletion count from the third-order tensor count model. An information processing device equipped with the following features.

2. The aforementioned information processing device is The information processing apparatus according to claim 1, comprising an inference unit that obtains a question text which is text representing a question, and infers an answer to the question based on part-of-speech words for the question obtained from the question text, the word count model, and the third-order tensor count model.

3. The inference unit, The information processing device according to claim 2, which generates a word-question count as a result of counting the part-of-speech words for the question, generates a second-order tensor count as a result of performing a second-order tensor count on the part-of-speech words for the question, and infers the answer to the question by calculating a PMI value based on the word-question count, the second-order tensor count, the word count model, and the third-order tensor count model.

4. The generating unit is The results of counting the parts of speech of the words obtained from the aforementioned pre-trained text are generated as an ngram count model. The aforementioned modification section is, The information processing device according to claim 3, which generates an ngram deletion count as the result of counting the part-of-speech words to be deleted, and subtracts the ngram deletion count from the ngram count model.

5. The inference unit, The information processing device according to claim 4, which infers the answer to the question sentence based on the word question count, the second-order tensor count, the ngram count model, the word count model, and the third-order tensor count model.

6. The aforementioned modification section is, An information processing apparatus according to any one of claims 3 to 5, comprising: generating a word addition count as a result of counting additional words with parts of speech obtained from additional text which is text to be added to the pre-learned text; generating a third-order tensor addition count as a result of performing a third-order tensor count on the additional words with parts of speech; adding the word addition count to the word count model; and adding the third-order tensor addition count to the third-order tensor count model.

7. The generating unit is From the tagged pre-trained text, the tagged word count model and the third-order tensor count model are generated. The aforementioned modification section is, The information processing apparatus according to claim 6, comprising generating a tagged word deletion count and a third-order tensor deletion count from the tagged deletion text, subtracting the word deletion count from the word count model using the tags, and subtracting the third-order tensor deletion count from the third-order tensor count model.

8. The aforementioned modification section is, The information processing apparatus according to claim 7, comprising generating a tagged word addition count and a third-order tensor addition count from the tagged additional text, adding the word addition count to the word count model using the tags, and adding the third-order tensor addition count to the third-order tensor count model.

9. The generating unit is From the aforementioned pre-training texts for each field, the aforementioned word count model and the aforementioned third-order tensor count model are generated for each field. The inference unit, An information processing device according to any one of claims 3 to 5, which obtains weights for the fields of the word count model and the third-order tensor count model, and uses the word count model and the third-order tensor count model according to the weights to infer the answer to the question.

10. Computers The results of counting the parts of speech words obtained from the pre-training text are generated as a word count model, and the results of performing a third-order tensor count on the parts of speech words obtained from the pre-training text are generated as a third-order tensor count model. An information processing method comprising: generating a word deletion count by counting the part-of-speech words to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text; generating a third-order tensor deletion count by performing a third-order tensor count on the part-of-speech words to be deleted; subtracting the word deletion count from the word count model; and subtracting the third-order tensor deletion count from the third-order tensor count model.

11. The generation process involves generating a word count model from the count of parts of speech words obtained from a pre-training text, and generating a third-order tensor count model from the result of performing a third-order tensor count on the parts of speech words obtained from the pre-training text. An information processing program that causes a computer to perform the following steps: generate a word deletion count by counting the words with parts of speech to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text; generate a third-order tensor deletion count by performing a third-order tensor count on the words with parts of speech to be deleted; subtract the word deletion count from the word count model; and perform a modification process by subtracting the third-order tensor deletion count from the third-order tensor count model.

Images