Information processing device, information processing method, and information processing program

Abstract

An information processing device (100) infers an answer to a query. A generation unit (110) generates a word count model (41) and a third-order tensor count model (42) by counting words with parts of speech obtained from a pre-learning text (21). A correction unit (120) generates a word deletion count (31) and a third-order tensor deletion count (32) by counting words with parts of speech to be deleted that are obtained from a deletion text (22). The correction unit (120) subtracts the word deletion count (31) from the word count model (41), and subtracts the third-order tensor deletion count (32) from the third-order tensor count model (42).

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] Large-scale language models (LLMs) are generated by pre-training from large corpora. LLM is an abbreviation for Large Language Models. However, from the perspectives of privacy protection, copyright infringement, bias issues, and data updates, there is a demand for unlearning (hereinafter referred to as unlearning), in which partial examples from a large-scale corpus are deleted after the large-scale language model has been generated (i.e., the pre-training is made to appear as if it had not been performed). In conventional deep learning, if partial examples from a large-scale corpus are deleted, all pre-training must be redone. This poses the problem of taking a lot of time and being extremely costly.

[0003] Patent Literature 1 discloses a technique for dividing training data for a machine learning model into a first set that includes private data and a second set that does not include private data.

[0004] Japanese Patent Application Laid-Open No. 2021-193533

[0005] Okimura et al., "Attribute-Based Unlearning in Natural Language Processing," 30th Annual Conference of the Association for Natural Language Processing, 2024.

[0006] In Patent Document 1, a machine learning model is trained using a first set containing private data, and the machine learning model is divided into an input model and an output model. The output model is then trained using a second set. In this way, Patent Document 1 trains the output model by so-called transfer learning, excluding private information. This again poses the problem that training takes a long time and unlearning cannot be completely reversible. Various other unlearning methods for natural language processing are also described in Non-Patent Document 1, but completely reversible forgetting is not easy.

[0007] The present disclosure aims to perform partial unlearning from a pre-trained model with high accuracy in the sense of completely reversibly forgetting in a reasonable amount of time.

[0008] The information processing device according to the present disclosure includes: a generation unit that generates, as a word count model, the results of counting words with parts of speech obtained from a pre-training text, which is a text for pre-training, and generates, as a third-order tensor count model, the results of performing third-order tensor counting on the words with parts of speech obtained from the pre-training text; and a correction unit that generates, as word deletion counts, the results of counting words with parts of speech for deletion obtained from a deletion text, which is a text to be deleted from the pre-training text, and generates, as third-order tensor deletion counts, the results of performing third-order tensor counting on the words with parts of speech for deletion, and subtracts the word deletion counts from the word count model and subtracts the third-order tensor deletion counts from the third-order tensor count model.

[0009] The information processing device according to the present disclosure generates a count model consisting of counts, and subtracts the count generated from the deleted text from the count model. Thus, the information processing device according to the present disclosure has the advantage of being able to partially unlearn a pre-trained count model in a reasonable amount of time with high accuracy.

[0010] FIG. 1 is a diagram showing an example of the configuration of an information processing device according to embodiment 1. FIG. 2 is a flow diagram showing an example of model generation correction processing according to embodiment 1. FIG. 3 is a diagram showing a comparative example third-order tensor model to be compared with the word count model and the third-order tensor count model according to embodiment 1. FIG. 3 is a diagram showing an example of the configuration of an information processing device according to a modification of embodiment 1. FIG. 4 is a flow diagram showing an example of inference processing according to embodiment 2. FIG. 5 is a diagram showing a specific example of inference processing according to embodiment 2. FIG. 4 is a diagram showing an example of the configuration of an information processing device according to embodiment 3. FIG. 5 is a flow diagram showing an example of model generation correction processing according to embodiment 3. FIG. 6 is a diagram showing a specific example of inference processing according to embodiment 3. FIG. 6 is a flow diagram showing an example of model generation correction processing according to embodiment 4. FIG. 7 is a diagram showing an example of tagged text according to embodiment 5. FIG. 7 is a flow diagram showing an example of model generation correction processing according to embodiment 5. FIG. 8 is a flow diagram showing an example of model generation correction processing according to embodiment 6.

[0011] The present embodiment will be described below with reference to the drawings. In each drawing, the same or corresponding parts are assigned the same reference numerals. In the description of the embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. The arrows in the drawings mainly indicate the flow of data or the flow of processing.

[0012] Embodiment 1. ***Description of Configuration*** Fig. 1 is a diagram showing an example of the configuration of an information processing device 100 according to this embodiment. The information processing device 100 is a computer. The information processing device 100 includes a processor 910, as well as other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to the other hardware via signal lines and controls this other hardware.

[0013] The information processing device 100 includes, as functional elements, a generation unit 110, a correction unit 120, an inference unit 130, and a storage unit 140. A word count model 41 and a third-order tensor count model 42 are stored in the storage unit 140. The word count model 41 and the third-order tensor count model 42 may be collectively referred to as the count model 40.

[0014] The functions of the generating unit 110, the correcting unit 120, and the inferring unit 130 are realized by software. The storage unit 140 is provided in the memory 921. Note that the storage unit 140 may be provided in the auxiliary storage device 922, or may be provided separately in the memory 921 and the auxiliary storage device 922.

[0015] The processor 910 is a device that executes an information processing program. The information processing program is a program that realizes the functions of the generation unit 110, the correction unit 120, and the inference unit 130. The processor 910 is an IC that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP, and a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0016] The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The auxiliary storage device 922 is a storage device that saves data. A specific example of the auxiliary storage device 922 is an HDD. The auxiliary storage device 922 may also be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

[0017] The input interface 930 is a port connected to an input device such as a mouse, keyboard, or touch panel. Specifically, the input interface 930 is a USB terminal. The input interface 930 may also be a port connected to a LAN. USB is an abbreviation for Universal Serial Bus. LAN is an abbreviation for Local Area Network.

[0018] The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) terminal. Specifically, the display is an LCD. The output interface 940 is also called a display interface. HDMI (registered trademark) is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display.

[0019] The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, a telephone line, or Wi-Fi (registered trademark). Specifically, the communication device 950 is a communication chip or NIC. NIC is an abbreviation for Network Interface Card.

[0020] The information processing program is executed in the information processing device 100. The information processing program is read into the processor 910 and executed by the processor 910. The memory 921 stores not only the information processing program but also an OS. OS is an abbreviation for Operating System. The processor 910 executes the information processing program while executing the OS. The information processing program and the OS may be stored in an auxiliary storage device 922. The information processing program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the information processing program may be incorporated into the OS.

[0021] The information processing device 100 may include multiple processors that replace the processor 910. These multiple processors share the task of executing the information processing program. Each processor is a device that executes the information processing program in the same way as the processor 910.

[0022] Data, information, signal values ​​and variable values ​​used, processed or output by the information processing program are stored in the memory 921, the auxiliary storage device 922, or a register or cache memory within the processor 910.

[0023] The "parts" of the generating unit 110, the correcting unit 120, and the inference unit 130 may be read as "circuits," "steps," "procedures," "processing," or "circuits." The information processing program causes a computer to execute a generating process, a correcting process, and an inference process. The "processing" of the generating process, the correcting process, and the inference process may be read as a "program," a "program product," a "computer-readable storage medium storing a program," or a "computer-readable recording medium recording a program." Furthermore, the information processing method is a method performed by the information processing device 100 executing an information processing program. The information processing program may be provided by being stored in a computer-readable recording medium. Furthermore, the information processing program may be provided as a program product.

[0024] 1 shows an example of a configuration of the information processing device 100 in which a model generation / modification function for generating and modifying a model and an inference phase for performing inference using the model are implemented in a single device. In other configuration examples, the model generation / modification function and the inference function may be installed in different devices. For example, the information processing device 100 may be a system including a first device and a second device. The information processing device 100 may be configured as an information processing system in which the first device is equipped with a generation unit and a modification unit, and the second device is equipped with an inference unit.

[0025] ***Description of Operation*** Next, the operation of the information processing device 100 according to this embodiment will be described. The operating procedure of the information processing device 100 corresponds to an information processing method. Furthermore, a program that realizes the information processing, which is the operation of the information processing device 100, corresponds to an information processing program. The information processing, which is the operation of the information processing device 100, comprises a model generation correction process that realizes a model generation correction function, and an inference process that realizes an inference function. In this embodiment, the model generation correction process will be described, and the inference process corresponding to the model generation correction process according to this embodiment will be described in embodiment 2.

[0026] <Model Generation and Correction Processing> Fig. 2 is a flow diagram showing an example of the model generation and correction processing according to this embodiment. The model generation and correction processing includes a generation processing and a correction processing.

[0027] <<Generation Process: Steps S11 to S14>> In the generation process, the generation unit 110 counts words with parts of speech obtained from the pre-training text 21 and generates the result as a word count model 41. The generation unit 110 also performs third-order tensor counting on the words with parts of speech obtained from the pre-training text 21 and generates a third-order tensor count model 42. Specifically, this is as follows.

[0028] In step S11, the generation unit 110 acquires a pre-training text 21, for example, via the input interface 930. The pre-training text 21 is a text used for pre-training for model generation. The generation unit 110 performs morphological analysis on the pre-training text 21 to acquire words with parts of speech. The words with parts of speech are a string of multiple words, each of which has a part of speech assigned to it.

[0029] In step S12, the generation unit 110 counts words with parts of speech obtained from the pre-training text 21. In step S13, the generation unit 110 performs third-order tensor counting on the words with parts of speech obtained from the pre-training text 21. In step S14, the generation unit 110 generates a count model 40. Specifically, the generation unit 110 stores the results of the counting in step S12 in the storage unit 140 as a word count model 41. The word count model 41 is frequency information for each word with a part of speech. In addition, the generation unit 110 stores the results of the third-order tensor counting in step S13 in the storage unit 140 as a third-order tensor count model 42. The third-order tensor count model 42 is frequency information for each of three words, each of which consists of one predicate (what to do) and two arguments (subject, object, etc.).

[0030] <<Correction Process: Steps S21 to S24>> In the correction process, the correction unit 120 counts the parts-of-speech-associated words for deletion obtained from the deleted text 22 and generates the result as the word deletion count 31. The correction unit 120 also performs third-order tensor counting on the parts-of-speech-associated words for deletion and generates the result as the third-order tensor deletion count 32. The correction unit 120 then subtracts the word deletion count 31 from the word count model 41, and subtracts the third-order tensor deletion count 32 from the third-order tensor count model 42. Specifically, this is as follows.

[0031] In step S21, the correction unit 120 acquires the deletion text 22, for example, via the input interface 930. The deletion text 22 is text to be deleted from the pre-training text 21. The correction unit 120 performs morphological analysis on the deletion text 22 to acquire words with parts of speech to be deleted. As shown in FIG. 2 , the words with parts of speech to be deleted are a string of multiple words, each of which is associated with a part of speech.

[0032] In step S22, the correction unit 120 counts the words with parts of speech to be deleted and generates the result as the word deletion count 31. The word deletion count 31 is frequency information of each word with parts of speech to be deleted.

[0033] In step S23, the correction unit 120 performs third-order tensor counting on the part-of-speech-attached words to be deleted, and generates the result as third-order tensor deletion count 32. The third-order tensor deletion count 32 is frequency information for each of three words consisting of one predicate word (what to do) and two terms (subject, object, etc.).

[0034] In step S24, correction unit 120 subtracts word deletion count 31 from word count model 41, and subtracts third-order tensor deletion count 32 from third-order tensor count model 42. Correction unit 120 stores word count model 41 after the subtraction and third-order tensor count model 42 after the subtraction in storage unit 140.

[0035] ***Explanation of the Effects of the Present Embodiment*** FIG. 3 is a diagram showing a comparative third-order tensor model for comparison with the word count model 41 and the third-order tensor count model 42 according to the present embodiment. The upper part of FIG. 3 shows an example of generating the comparative third-order tensor model. The comparative third-order tensor model is generated from a word count, which is the result of counting words with parts of speech obtained from the pre-training text 21, and a third-order tensor count, which is the result of performing a third-order tensor count on the words with parts of speech obtained from the pre-training text 21. Specifically, the comparative third-order tensor model is a PMI value calculated using the word count and the third-order tensor count. The PMI value, PMI(v, x, y), is calculated using Equation 1 in the upper part of FIG. 3. P indicates the probability of each event. α is, for example, α=11. In this way, the PMI value (information content) is directly stored in the comparative third-order tensor model. Therefore, if only a portion of the training data is deleted from the pre-training text 21, the entire system must be retrained, i.e., the PMI value must be recalculated.

[0036] Here, PMI(v, x, y) is the mutual information that measures the degree of association between events v, x, and y. PMI is an abbreviation for Pointwise Mutual Information. In PMI(v, x, y), v is a single predicate (what to do), and x and y are two terms (subject, object, etc.).

[0037] The lower part of FIG. 3 shows an example of decomposing a third-order tensor model of the comparative example into a word count model 41 according to the present embodiment and a third-order tensor count model 42. As shown in the lower part of FIG. 3, the third-order tensor model PMI(w ｉ , w ｊ , Y ｋ ) is calculated by the word count model C(w ｉ ), C(w ｊ ), C(Y ｋ ) and the third-order tensor counting model C(w ｉ , w ｊ , Y ｋ ) where C indicates the count of each event, N is the total word frequency, and α=11.

[0038] In the information processing device 100 according to this embodiment, a PMI value is not calculated during learning, but a word count model 41 and a third-order tensor count model 42 are generated as a count model 40. As will be described in detail in a second embodiment, by generating the count model 40, a PMI value is calculated using the word count model 41 and the third-order tensor count model 42 during inference, and inference is performed.

[0039] The information processing device 100 according to this embodiment generates a word deletion count and a third-order tensor deletion count for the deleted text, and can subtract these counts from the word count model 41 and the third-order tensor count model 42. That is, the information processing device 100 according to this embodiment breaks down the input deleted text into counts and subtracts the counts to return only the deleted text to an untrained state. Therefore, the information processing device 100 according to this embodiment can quickly and accurately cancel the deleted text from a model obtained by training pre-trained text. Furthermore, P(v), P(x), and P(y), which are necessary for PMI calculation, all change when N is subtracted. Therefore, in a model having P(v), P(x), and P(y), all of the P(v), P(x), and P(y), must be recalculated when N is subtracted. On the other hand, if the model has C(v), C(x), and C(y), recalculation during model training is not required. The information processing device 100 according to this embodiment stores the sum in the word count model as N, enabling partial cancellation in a reasonable amount of time.

[0040] ***Other Configurations*** In this embodiment, the functions of the generation unit 110, the correction unit 120, and the inference unit 130 are realized by software. As a modification, the functions of the generation unit 110, the correction unit 120, and the inference unit 130 may be realized by hardware. Specifically, the information processing device 100 includes an electronic circuit 909 instead of the processor 910.

[0041] 4 is a diagram showing an example of the configuration of an information processing device 100 according to a modified example of this embodiment. The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the generation unit 110, the correction unit 120, and the inference unit 130. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0042] The functions of the generating unit 110, the correcting unit 120, and the inferring unit 130 may be realized by a single electronic circuit, or may be realized by distributing them among a plurality of electronic circuits.

[0043] As another modification, some of the functions of the generation unit 110, the correction unit 120, and the inference unit 130 may be implemented by electronic circuits, and the remaining functions may be implemented by software. Also, some or all of the functions of the generation unit 110, the correction unit 120, and the inference unit 130 may be implemented by firmware.

[0044] Each of the processor and the electronic circuit is also called a processing circuit, and the functions of the generating unit 110, the correcting unit 120, and the inferring unit 130 are realized by the processing circuit.

[0045] Embodiment 2 In this embodiment, the following mainly describes the points added to embodiment 1. In this embodiment, the same reference numerals are used to designate components having the same functions as those in embodiment 1, and the description thereof will be omitted.

[0046] ***Description of Configuration*** The configuration of the information processing device 100 according to this embodiment is the same as that of the first embodiment.

[0047] ***Description of Operation*** In this embodiment, the inference process will be described.

[0048] <Inference Processing> Fig. 5 is a flow diagram showing an example of the inference processing according to this embodiment. In the inference processing, the inference unit 130 acquires the question text 23, which is text representing a question. The inference unit 130 infers an answer to the question based on the question-specific words with parts of speech acquired from the question text 23, the word count model 41, and the third-order tensor count model 42. Specifically, the process is as follows.

[0049] In step S31, the inference unit 130 acquires the question text 23, for example, via the input interface 930. The question text 23 is, for example, text representing a question. In the example of FIG. 3, the question text 23 is "Where does Taro X live?" The inference unit 130 performs morphological analysis on the question text 23 to acquire words with parts of speech for the question. As shown in FIG. 5, the words with parts of speech for the question are a string of multiple words, each of which is associated with a part of speech.

[0050] In step S32, the inference unit 130 generates the result of counting the question words with parts of speech as the word question count 33. In step S33, the inference unit 130 performs second-order tensor counting on the question words with parts of speech and generates the result as the second-order tensor count 34. Specifically, the second-order tensor count 34 is frequency information for each of two words consisting of one predicate word (what to do) and one argument word.

[0051] In step S34, the inference unit 130 infers an answer to the question by calculating a PMI value based on the word question count 33, the second-order tensor count 34, the word count model 41, and the third-order tensor count model 42. Note that the inference unit 130 may infer an answer to the question by calculating a PMI value based on the second-order tensor count 34, the word count model 41, and the third-order tensor count model 42. In step S35, the inference unit 130 outputs a word string of answer word candidates obtained as a result of the inference as an inference result 35. The word strings of the answer word candidates may be ranked by likelihood indicating the likelihood and output.

[0052] Specifically, the inference unit 130 calculates the PMI value of the word candidate in the following procedure: (1) Generate all words from the word count model 41. (2) Generate a search key for a third-order tensor from (1) and the second-order tensor count 34. (3) Search for the count number from the third-order tensor count model 42 using the search key from (2). (4) Calculate the PMI value of the answer word candidate using the total number of words, the word frequency, and (3).

[0053] FIG. 6 is a diagram showing a specific example of the inference process according to this embodiment. ｉ , w ｊ , Y ｋ ) in which Y ｋ As explained in (Equation 2) of the first embodiment, PMI(w ｉ , w ｊ , Y ｋ ) can be expressed by the count numbers of the word count model 41 and the third-order tensor count model 42. The inference unit 130 receives the PMI value of the answer word candidate as an input and calculates Y ｋ The likelihood L(Y ｋ ) is calculated. ｋ The likelihood L(Y ｋ ) is expressed by (Equation 3). Then, the inference unit 130 uses (Equation 4) to calculate Y ｋ The likelihood L(Y ｋ ) is equal to or greater than the threshold. ｋ The likelihood L(Y ｋ ) is equal to or greater than a threshold, the answer word candidates may be output in order of likelihood. If the likelihood of all the candidates does not meet the threshold, the inference unit 130 may output a message such as "No information available."

[0054] In the example of FIG. 6, in response to the question text “What animals can you keep at home?”, the inference result 35 is Y １ From Y ５ The answer word candidates are output. １ From Y ５ The answer word candidates may be ranked by likelihood and output.

[0055] ***Description of Effects of the Present Embodiment*** In the information processing device 100 according to the present embodiment, a word count model and a third-order tensor count model are generated as count models during learning, without calculating PMI values. During inference, the word count model and the third-order tensor count model are used to calculate PMI values ​​of answer word candidates and infer answer words for a question. Therefore, the information processing device 100 according to the present embodiment has the effect of being able to infer an answer to a question using a count model in which deleted text has been cancelled with high accuracy in a short amount of time.

[0056] Embodiment 3 In this embodiment, differences from and additions to embodiments 1 and 2 will be mainly described. In this embodiment, components having the same functions as those in embodiments 1 and 2 will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0057] 7 is a diagram showing an example of the configuration of information processing device 100 according to this embodiment. In addition to the configuration described in embodiment 1, information processing device 100 according to this embodiment stores ngram count model 43 in storage unit 140.

[0058] ***Description of Operation*** Figure 8 is a flow diagram showing an example of a model generation and correction process according to this embodiment. Figure 8 explains the generation and correction process of an ngram count model 43 that is newly added to the model generation and correction process described in embodiment 1. The generation and correction process of the word count model 41 shown in Figure 8 is the same as in embodiment 1. Also in this embodiment, a third-order tensor count model 42 is generated and corrected, but as this is the same as in embodiment 1, a description thereof will be omitted.

[0059] <Model Generation and Correction Process> <<Generation Process>> In step S43, the generation unit 110 performs ngram counting on words with parts of speech attached obtained from the pre-training text 21. Ngrams are a technique for dividing a portion of a sentence into sequences of any number n of words. Here, frequency information on ngrams sequenced with any number n of words is obtained for words with parts of speech attached obtained from the pre-training text 21. In step S44, the generation unit 110 generates an ngram count model 43 based on the results of counting ngrams on words with parts of speech attached obtained from the pre-training text 21, and stores the result in the storage unit 140. The ngram count model 43 is ngram frequency information for words with parts of speech attached obtained from the pre-training text 21.

[0060] <<Correction Process>> In step S53, the correction unit 120 counts ngrams for the part-of-speech-associated word to be deleted and generates the result as the ngram deletion count 36. The ngram deletion count 36 is frequency information of ngrams for the part-of-speech-associated word to be deleted. In step S54, the correction unit 120 subtracts the ngram deletion count 36 from the ngram count model 43.

[0061] 9 is a flow diagram showing an example of inference processing according to the present embodiment. In FIG. 9, an inference processing using an ngram count model 43 newly added to the inference processing described in the first embodiment is described.

[0062] Inference Process In step S63, the inference unit 130 performs second-order tensor counting on the part-of-speech-associated words for the question to generate second-order tensor counts 34. The inference unit 130 also performs ngram counting on the part-of-speech-associated words for the question to generate ngram question counts 39. The ngram question counts 39 are ngram frequency information for the part-of-speech-associated words for the question.

[0063] In step S64, the inference unit 130 infers an answer to the question based on the word question count 33, the second-order tensor count 34, the ngram question count 39, the ngram count model 43, the word count model 41, and the third-order tensor count model 42. In step S65, the inference unit 130 outputs, as an inference result 35, a word string of answer word candidates obtained as a result of the inference.

[0064] Specifically, the inference unit 130 calculates the PMI value of the word candidate using the following procedure: (1) Generate all words from the word count model 41. (2) Generate a search key for a third-order tensor from (1) and the second-order tensor count 34. (3) Search for the count number of the third-order tensor from the third-order tensor count model 42 using the search key from (2). (4) Calculate the PMI value of the answer word candidate using the total number of words, word frequency, and (3). (5) Calculate the likelihood by weighting the ngram using the ngram question count 39 and the ngram count model 43.

[0065] Fig. 10 is a diagram showing a specific example of the inference process according to this embodiment. The bold square frame in Fig. 10 indicates the probability P of an ngram. The dotted square frame in Fig. 10 is an equation expressing the probability P of an ngram in terms of the count C of the ngram. In this way, ngrams can also be decomposed into counts and stored.

[0066] By decomposing the probability of ngrams into counts and storing them, a language model can be constructed, enabling prediction of word candidates for any position while enabling unlearning. ｎ At this time, ngram probability is replaced with ngram and n-1gram counts for learning. (b) Backward ngram ｎ At this time, ngram probability is replaced with ngram and n-1gram counts for learning. (c) (Equation 5) is the weighted sum of predicate term PMI for the AND candidates of (a) and (b), and W ｎ Here, β is a weighting coefficient.

[0067] ***Description of Effects of the Present Embodiment*** As described above, according to the information processing device 100 of the present embodiment, in addition to the functions of the first and second embodiments, an ngram count model can be used as a count model. Furthermore, according to the information processing device 100 of the present embodiment, word counts and ngram counts can be calculated from deleted text, and the word count model and ngram count model can be modified. In this way, ngram probability values ​​are not directly stored in the model database, but are decomposed and stored as ngram count models into ngrams (2-grams) and word counts (unigram counts).

[0068] Therefore, according to the information processing device 100 of this embodiment, it is possible to cancel deleted text with high accuracy in a short time for the ngram count model. Furthermore, according to the information processing device 100 of this embodiment, since it is possible to use the ngram count model, it is possible to achieve an effect of performing inference with higher accuracy.

[0069] Embodiment 4 In this embodiment, the following will be mainly described: points that are added to Embodiments 1 to 3. In this embodiment, components having the same functions as those in Embodiments 1 to 3 are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0070] *** Description of Configuration *** The configuration of the information processing device 100 according to this embodiment is the same as that of Embodiment 1. In Embodiment 1, an aspect in which a portion of a pre-training text that is desired to be deleted is removed from the model has been described. In this embodiment, an aspect in which a portion of a pre-training text that is desired to be added to the count model will be described.

[0071] ***Description of Operation*** Figure 11 is a flow diagram showing an example of the model generation correction process according to this embodiment. In this embodiment, the deleted text 22 described in embodiment 1 is replaced with the added text 24 to be added to the pre-learning text 21. Other points are the same as in embodiment 1. The generation process within the model generation correction process is the same as in embodiment 1. The inference process is also the same as in embodiment 1. In the correction process within the model generation correction process, the correction unit 120 acquires the added text 24 and performs the same process as in embodiment 1.

[0072] In step S21, the correction unit 120 obtains additional text 24, which is text to be added to the pre-training text 21. The correction unit 120 obtains additional words with parts of speech by performing morphological analysis on the additional text 24. In step S22, the correction unit 120 counts the additional words with parts of speech and generates the result as word additional count 37. In step S23, the correction unit 120 performs third-order tensor counting on the additional words with parts of speech and generates the result as third-order tensor additional count 38. In step S24, the correction unit 120 adds the word additional count 37 to the word count model 41 and adds the third-order tensor additional count 38 to the third-order tensor count model 42.

[0073] ***Explanation of the effect of this embodiment*** According to the information processing device 100 of this embodiment, it is possible to add and reflect events that you want to add to the pre-training data to the count model with high accuracy in a reasonable amount of time.

[0074] Embodiment 5 In this embodiment, the following will be mainly described: points that are added to Embodiments 1 to 4. In this embodiment, the same reference numerals are used to designate components having the same functions as those in Embodiments 1 to 4, and descriptions thereof will be omitted.

[0075] *** Description of Configuration *** The configuration of the information processing device 100 according to this embodiment is the same as that of embodiment 1. In this embodiment, the pre-learning text 21 is tagged text. The deleted text 22 and the added text 24 are extracted from the tagged text.

[0076] FIG. 12 is a diagram showing an example of tagged text according to this embodiment. The tagged text shown in FIG. 12 is a C4(ja) corpus. The text is given tag information such as a URL or a timestamp. By using such tagged text, it is possible to specify, for example, a target period for deletion or addition, and to eliminate old text.

[0077] ***Description of Operation*** Figure 13 is a flow diagram showing an example of a model generation and correction process according to this embodiment. The model generation and correction process and the inference process are the same as those in embodiment 1. In step S01, a pre-training text 21 is extracted from the pre-training tagged text. In step S02, a deleted text 22 and an added text 24 are extracted from the pre-training tagged text or other tagged text.

[0078] In FIG. 13, deleted text 22 and added text 24 are input for the correction process, and the correction process for each of the deleted text 22 and added text 24 is the same as in the first and fourth embodiments.

[0079] The generation unit 110 generates a tagged word count model 41 and a third-order tensor count model 42 from tagged pre-training text 21. The correction unit 120 generates tagged word deletion counts 31 and third-order tensor deletion counts 32 from tagged deletion text 22. The correction unit 120 subtracts the word deletion counts 31 from the word count model 41 using the tags. The correction unit 120 also subtracts the third-order tensor deletion counts 32 from the third-order tensor count model 42 using the tags. The correction unit 120 generates tagged word addition counts 37 and third-order tensor addition counts 38 from tagged additional text 24. The correction unit 120 adds the word addition counts 37 to the word count model 41 using the tags. The correction unit 120 also adds the third-order tensor addition counts 38 to the third-order tensor count model 42 using the tags.

[0080] ***Explanation of Effect of This Embodiment*** According to the information processing device 100 of this embodiment, it is possible to achieve the effect that the learning contents of the model can be freely edited by using tags.

[0081] Embodiment 6 In this embodiment, the following mainly describes the points added to Embodiments 1 to 5. In this embodiment, the same reference numerals are used to designate components having the same functions as those in Embodiments 1 to 5, and the description thereof will be omitted.

[0082] *** Description of Configuration *** The configuration of the information processing device 100 according to this embodiment is the same as that of embodiment 1. In this embodiment, as in embodiment 5, the pre-learning text 21 is tagged text, and the deleted text 22 and the added text 24 are extracted from the tagged text. The tag is assumed to include the domain type. For example, the text is assumed to be classifiable into fields such as art, science, or politics.

[0083] In this embodiment, a word count model 41 and a third-order tensor count model 42 are generated for each domain of the pre-training text 21.

[0084] ***Description of Operation*** Figure 14 is a flow diagram showing an example of inference processing according to this embodiment. In the generation process of the model generation and correction process, pre-training text 21 is processed for each domain, and a word count model 41 and a third-order tensor count model 42 are generated for each domain. In the correction process of the model generation and correction process, deleted text 22 or added text 24 is processed for each domain, and subtraction or addition is performed on word count model 41 and third-order tensor count model 42 for each domain. In Figure 14, a word count model 41 and a third-order tensor count model 42 are generated for each of domains 1, 2, and 3.

[0085] <Inference Process> In step S74, the inference unit 130 obtains weights for each field of the word count model 41 and the third-order tensor count model 42. The inference unit 130 infers an answer to the question using the word count model and the third-order tensor count model according to the weights for the field. The inference unit 130 may obtain the weights via the input interface 930. Alternatively, the inference unit 130 may have a function that allows selection of weights based on predetermined conditions.

[0086] ***Explanation of Effects of the Present Embodiment*** According to the information processing device 100 of the present embodiment, it is possible to achieve the effect of switching the learning contents of models in a plurality of fields and changing the weighting.

[0087] In the above first to sixth embodiments, each unit of the information processing device has been described as an independent functional block. However, the configuration of the information processing device does not have to be the same as that of the above-described embodiments. The functional blocks of the information processing device may have any configuration as long as they can realize the functions described in the above-described embodiments. As described above, the information processing device may not be a single device, but may be a system composed of multiple devices. Furthermore, multiple parts of the first to sixth embodiments may be combined and implemented. Alternatively, only one part of these embodiments may be implemented. In addition, these embodiments may be combined in any way, either as a whole or in part. That is, in the first to sixth embodiments, each embodiment may be freely combined, or any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0088] The above-described embodiments are essentially preferred examples and are not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, or the scope of use of the present disclosure. The above-described embodiments can be modified in various ways as needed. For example, the procedures described using flow charts or sequence diagrams may be modified as appropriate.

[0089] 21 pre-training text, 22 deleted text, 23 question text, 24 added text, 31 word deletion count, 32 third-order tensor deletion count, 33 word question count, 34 second-order tensor count, 35 inference result, 36 ngram deletion count, 37 word addition count, 38 third-order tensor addition count, 39 ngram question count, 40 count model, 41 word count model, 42 third-order tensor count model, 43 ngram count model, 100 information processing device, 110 generation unit, 120 correction unit, 130 inference unit, 140 storage unit, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface, 940 output interface, 950 communication device.

Claims

1. A generation unit generates a word count model from the results of counting parts of speech words obtained from a pre-training text, which is a pre-training text, and generates a third-order tensor count model from the results of performing a third-order tensor count on the parts of speech words obtained from the pre-training text. The modification unit generates a word deletion count by counting the words with parts of speech to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text, and generates a third-order tensor deletion count by performing a third-order tensor count on the aforementioned words with parts of speech to be deleted, subtracts the word deletion count from the word count model, and subtracts the third-order tensor deletion count from the third-order tensor count model. An information processing device equipped with the following features.

2. The aforementioned information processing device is The information processing apparatus according to claim 1, comprising an inference unit that obtains a question text which is text representing a question, and infers an answer to the question based on part-of-speech words for the question obtained from the question text, the word count model, and the third-order tensor count model.

3. The inference unit, The information processing device according to claim 2, which generates a word-question count as a result of counting the part-of-speech words for the question, generates a second-order tensor count as a result of performing a second-order tensor count on the part-of-speech words for the question, and infers the answer to the question by calculating a PMI value based on the word-question count, the second-order tensor count, the word count model, and the third-order tensor count model.

4. The generating unit is The results of counting the parts of speech of the words obtained from the aforementioned pre-trained text are generated as an ngram count model. The aforementioned modification section is, The information processing device according to claim 3, which generates an ngram deletion count as the result of counting the part-of-speech words to be deleted, and subtracts the ngram deletion count from the ngram count model.

5. The inference unit, The information processing device according to claim 4, which infers the answer to the question sentence based on the word question count, the second-order tensor count, the ngram count model, the word count model, and the third-order tensor count model.

6. The aforementioned modification section is, An information processing apparatus according to any one of claims 3 to 5, comprising: generating a word addition count as a result of counting additional words with parts of speech obtained from additional text which is text to be added to the pre-learned text; generating a third-order tensor addition count as a result of performing a third-order tensor count on the additional words with parts of speech; adding the word addition count to the word count model; and adding the third-order tensor addition count to the third-order tensor count model.

7. The generating unit is From the tagged pre-trained text, the tagged word count model and the third-order tensor count model are generated. The aforementioned modification section is, The information processing apparatus according to claim 6, comprising generating a tagged word deletion count and a third-order tensor deletion count from the tagged deletion text, subtracting the word deletion count from the word count model using the tags, and subtracting the third-order tensor deletion count from the third-order tensor count model.

8. The aforementioned modification section is, The information processing apparatus according to claim 7, comprising generating a tagged word addition count and a third-order tensor addition count from the tagged additional text, adding the word addition count to the word count model using the tags, and adding the third-order tensor addition count to the third-order tensor count model.

9. The generating unit is From the aforementioned pre-training texts for each field, the aforementioned word count model and the aforementioned third-order tensor count model are generated for each field. The inference unit, An information processing device according to any one of claims 3 to 5, which obtains weights for the fields of the word count model and the third-order tensor count model, and uses the word count model and the third-order tensor count model according to the weights to infer the answer to the question.

10. Computers The results of counting the parts of speech words obtained from the pre-training text are generated as a word count model, and the results of performing a third-order tensor count on the parts of speech words obtained from the pre-training text are generated as a third-order tensor count model. An information processing method comprising: generating a word deletion count by counting the part-of-speech words to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text; generating a third-order tensor deletion count by performing a third-order tensor count on the part-of-speech words to be deleted; subtracting the word deletion count from the word count model; and subtracting the third-order tensor deletion count from the third-order tensor count model.

11. The generation process involves generating a word count model from the count of parts of speech words obtained from a pre-training text, and generating a third-order tensor count model from the result of performing a third-order tensor count on the parts of speech words obtained from the pre-training text. An information processing program that causes a computer to perform the following steps: generate a word deletion count by counting the words with parts of speech to be deleted obtained from the deletion text, which is the text to be deleted from the aforementioned pre-trained text; generate a third-order tensor deletion count by performing a third-order tensor count on the words with parts of speech to be deleted; subtract the word deletion count from the word count model; and perform a modification process by subtracting the third-order tensor deletion count from the third-order tensor count model.