Composite data generation system, method, user device, and program

Abstract

In the present invention, each of a plurality of user devices 1j includes: an output data acquisition unit 1j1 that obtains output data by inputting input data into a large-scale language model included in each user device; and a first transmission unit 1j2 that transmits the output data to an aggregation device 2. The aggregation device 2 includes: a determination unit 21 that determines which output data among the output data received from the plurality of user devices is to be transmitted to which user device among the plurality of user devices; and a second transmission unit 22 that transmits the determined output data to the determined user device. Each of the plurality of user devices 1j further includes an update unit 1j3 that, if there is output data received from the aggregation device 2, updates the input data using the output data received from the aggregation device 2. The plurality of user devices 1j and the aggregation device 2 repeat the process using the updated input data instead of the input data.

Description

Synthetic data generation system, method, user device, and program

[0001] The disclosed technology relates to a technology for generating synthetic data.

[0002] In recent years, by using large language models (LLMs) based on deep learning technology using neural network information processing, users have been able to artificially generate sentences.

[0003] The user inputs instructions and words or partial words (called tokens) to the LLM in the order X1, X2, ..., X i-1 A partial word is an element that makes up a word. In this case, the LLM can predict the next i-th token based on the token order co-occurrence probability and instructions. By repeating this process recursively, it is also possible to predict a sentence consisting of a token sequence. Here, the order co-occurrence probability is assumed to have been learned in advance using deep learning.

[0004] The specific effect of the prediction is X i Specific data of Z i ∈V (V is a candidate set of tokens predicted by the LLM, which is specific to the LLM and is predetermined), then X i =Z i The probability that Pr[X i =Z i |X i-1 ,…,X1] and present the most probable token candidates. i By adding the new token to the input, LLM will receive the next token, X i+1 By repeating the above process, LLM can generate highly likely sentences based on probability. LLM can also improve the accuracy of prediction by updating the order co-occurrence probability during the prediction process.

[0005] On the other hand, as the use of LLMs becomes more common, how to protect the privacy of the users who generate the text has become an issue. By analyzing the text output from LLMs backward, it is possible to extract the input tokens and token sequences X1,…,X i If it is possible to estimate these words and sentences with a high degree of certainty, third parties will be able to infer information about the user from these words, sentences, and the relationships between them, and in some cases may even be able to identify the user themselves.

[0006] In order to solve the above problems, the technology of Non-Patent Document 1 has been proposed as a method for protecting user privacy in LLM (see, for example, Non-Patent Document 1).

[0007] The method described in Non-Patent Document 1 extends the concept of differential privacy, which has been studied in the field of confidential statistical analysis, to the field of natural language processing, thereby ensuring the statistical security of data related to input sentence information in LLMs in the sense of differential privacy. According to this method, even though LLMs ultimately generate specific output sentences, they can incorporate appropriate ambiguity into the specific data of each input sentence that formed the basis of the output sentence. LLMs generate output sentences not only based on the individual data of the input sentences, but also on the semantic relationships between a large number of input sentences. Therefore, even if there is acceptable variation in the data of each input sentence, the variation has almost no effect on the final output sentence. This method exploits this characteristic unique to LLMs. Because input sentence information inherently varies, even if a third party attempts to identify the input sentence information data from the output sentence, they can only make an estimate within the range of variation. This protects the privacy of LLM users.

[0008] The technology of Non-Patent Document 1 will be referred to as the prior art below for comparison with the present invention. An overview of the prior art will be described below.

[0009] To simplify the explanation, we will use the example of a text classification problem. Here, we will classify each category in order as y1,…,y K Let D be the set of sentences that belong to those categories.(y1) ,…,D (yK) For example, category y i If the label of is "world", then D (yi) ={"Australia boosts antiterror measures at small air ports SYDNEY", "TOKYO - XXX Corp. is banking on the $3 billion deal to acquire", …}. In this way, multiple different sentences are associated with one category label. K is a predetermined positive integer.

[0010] The purpose of the prior art is to find the i-th sentence set D (yi) A set of input data that allows the user to input the sentences while retaining the LLM's sentence generation function and incorporating fluctuations into the output. (yi) D (yi) In the prior art, the user device is abbreviated as "user."

[0011] (Prior art / Procedure 1) The user receives label y i The initial set of data D (yi) ={z1,z2,…,z ni} where n i is the label y i This is the number of data items belonging to the group, and is assumed to be a sufficiently large value in the prior art.

[0012] (Prior art / Procedure 2) There are n users i Randomly select MN pieces of data from the data and create a sample set D consisting of N pieces of data. (yi),(m) ={z1 (m) ,z2 (m) ,…,z N (m)}, m∈[M] are prepared, where the integer m takes a value from 1 to M, corresponding to each of the M different sample sets.

[0013] (Prior art / Procedure 3) The user inputs the instructions and the label y prepared in Procedure 2. i A sample set D of(yi),(m) and the input information X 1:t is input into the LLM to obtain the probability distribution Pr[X|D (yi),(m) , Instruction]. Input information X 1:t is X1,X2,…,X t The input information so far is X 1:t is initialized to an empty string at t=1. The probability distribution Pr[X|D (yi),(m) ,Instruction] is a |V|-dimensional discrete probability distribution. This is done for all M different sample sets, m=1,...,M.

[0014] (Prior Art / Procedure 4) A user selects M probability distributions Pr[X|D (yi),(m) ,Instruction], Gaussian noise with a mean of zero and a variance of σ is added to each dimension to further modulate the probability values, and the final probability distribution Pr[~X|D (yi),(m) , Instruction] is calculated. Details of the calculation are described in Non-Patent Document 1. Then, the final probability distribution Pr[~X|D (yi),(m) ,Instruction], argmax sampling process (process to select token with maximum probability) is performed. This is X t+1 Then, repeat step 3. Repeating this process t+1≦T times yields one composite sentence ~z1, where T is a predetermined positive integer.

[0015] (Prior art / procedure 5) Furthermore, in procedures 2 to 4, the generation of tokens of length T from an empty string is performed by m i By repeating this process, the synthetic data set ~D (yi) = {~z1,…,~z mi}. In this way, we can obtain a set of sentences generated by LLM. T is a predetermined positive integer. m i is a predetermined positive integer.

[0016] The larger the number of sample sets M in step 2, the more diverse predictions by LLM can be incorporated into the synthesis of input data. Also, the larger the number of data N included in the sample set, the greater the fluctuations that can be included. Thus, the conventional method is effective in protecting the privacy of LLM users when a sufficiently large amount of data can be prepared for each label.

[0017] Xinyu Tang et.al, "PRIVACY-PRESERVING IN-CONTEXT LEARNING WITH DIFFERENTIALLY PRIVATE FEW-SHOT GENERATION", Conference Paper at ICLR 2024

[0018] However, the input sentence category y i When the number of initial data held by the user is small or the size of data that can be input to the machine learning system is small, it becomes difficult to adequately protect the user's privacy with the conventional method. This is because the user has only a set of all data D (yi) For a sufficiently large number of different sample sets D (yi),(m) ={z1 (m) ,z2 (m) ,…,z N (m)}, m∈[M], which narrows the range of inference a third party can make about the input data.

[0019] The disclosed technology aims to provide a synthetic data generation device, method, and program that can protect privacy even when a user device has a small amount of initial data.

[0020] One aspect of the disclosed technology is a synthetic data generation system including a plurality of user devices and an aggregation device, wherein each of the plurality of user devices includes an output data acquisition unit that obtains output data by inputting input data into a large-scale language model that the user device has, and a first transmission unit that transmits the output data to the aggregation device, and the aggregation device includes a determination unit that determines which output data, among the output data received from the plurality of user devices, to send to which of the plurality of user devices, and a second transmission unit that transmits the determined output data to the determined user device, and each of the plurality of user devices further includes an update unit that, if output data is received from the aggregation device, updates the input data using the output data received from the aggregation device, and the plurality of user devices and the aggregation device repeat the processing of the output data acquisition unit, first transmission unit, determination unit, second transmission unit, and update unit using the updated input data instead of the input data, and the updated input data at the end of the repeated processing is used as synthetic data.

[0021] According to the disclosed technology, privacy can be protected even when the number of pieces of initial data held by a user device is small.

[0022] Fig. 1 is a diagram showing an example of the functional configuration of a synthetic data generation system. Fig. 2 is a diagram showing an example of the functional configuration of a synthetic data generation device. Fig. 3 is a diagram showing an example of a processing procedure of a synthetic data generation method. Fig. 4 is a diagram showing an example of the functional configuration of a computer.

[0023] 1, the synthetic data generation system includes a plurality of user devices 11, ..., 1J and an aggregation device 2. The plurality of user devices 11, ..., 1J and the aggregation device 2 are connected via a network such as the Internet so that data can be transmitted and received. The communication path between the plurality of user devices 11, ..., 1J and the aggregation device 2 may be kept confidential by means of encryption or the like.

[0024] The synthetic data generation system assumes the use of J user devices, where J is a predetermined positive integer equal to or greater than 2. The jth user device will be identified by the reference symbol 1j.

[0025] As shown in FIG. 2, the user device 1j includes, for example, an output data acquisition unit 1j1, a first transmission unit 1j2, and an update unit 1j3.

[0026] As shown in FIG. 1 , the aggregation device 2 includes, for example, a determination unit 21 and a second transmission unit 22 .

[0027] The synthetic data generation method is realized by each unit of the synthetic data generation system performing at least one of the following steps S11 to S13. Fig. 3 shows an example of the processing procedure of the synthetic data generation method.

[0028] Note that the symbols "^" and "~" used in a sentence should normally be written directly above the character immediately following, but due to limitations in text notation, they are written immediately before the character in question. In mathematical formulas, these symbols are written in their proper position, i.e., directly above the character. For example, "~X" in a sentence is written as follows in a mathematical formula: Each part of the synthetic data generation system will be described below.

[0029] <Step S11> The output data acquisition unit 1j1 of the user device 1j, where j = 1, ..., J, inputs the input data into its own large-scale language model LLM to obtain output data (step S11). The output data is output to the first transmission unit 1j2.

[0030] For example, the output data acquisition unit 1j1 can obtain the output data by the methods (Prior Art / Procedure 1) to (Prior Art / Procedure 5) described above.

[0031] That is, the output data acquisition unit 1j1 first obtains the input sentence category y according to (Prior Art / Procedure 1). i Data set D j (yi) ={z1,z2,…,z ni (j)} is prepared. Here, n i (j) is the category y owned by user j i is the number of data items that belong to n i (j) is a predetermined positive integer. i (j)does not have to be a sufficiently large value.

[0032] i-th input sentence category y i is assumed to be common to all user devices j. Also, it is assumed that the instructions are common to all user devices 11, ..., 1J.

[0033] After that, the output data acquisition unit 1j1 (yi) ={z1,z2,…,z ni} instead of D j (yi) ={z1,z2,…,z ni (j) By processing (Prior Art / Procedure 2) to (Prior Art / Procedure 5) using}, a synthetic data set ~D j (yi) An example of input data is the dataset D j (yi) and the output data example is the synthetic data set ~D j (yi) is.

[0034] <Step S12> The output data is input to the first transmission unit 1j2.

[0035] For j=1, . . . , J, the first transmission unit 1j2 of the user device 1j transmits the input output data to the aggregation device 2 (step S12).

[0036] <Step S21> The determination unit 21 of the aggregation device 2 determines which output data received from the multiple user devices 11, ..., 1J to send to which user device among the multiple user devices 11, ..., 1J (step S21).

[0037] The output data received from the plurality of user devices 11, ..., 1J is ~D1 (yi) ,…,~D J (yi) If so, the determining unit 21 determines that ∼D1 (yi) ,…,~D J (yi) Randomly permute the columns of the synthetic data set after permute. (yi) ,…,^D J (yi)In this case, for example, the determination unit 21 determines ^D j (yi) The user terminal 1j then decides to transmit the request to the user terminal 1j.

[0038] As in this example, the determination unit 21 may randomly determine which output data received from multiple user devices 11, ..., 1J to send to which user device among the multiple user devices 11, ..., 1J.

[0039] <Step S22> The second transmission unit 22 of the aggregation device 2 transmits the output data determined by the determination unit 21 to the user device determined by the determination unit 21 (step S22).

[0040] This often results in a user device 1j obtaining a composite data set generated by a different user device 1j' (j'≠j).

[0041] <Step S13> For j = 1, ..., J, if there is output data received from the aggregation device 2, the update unit 1j3 of the user device 1j updates the input data using the output data received from the aggregation device 2 (step S13).

[0042] For example, if the output data received from the aggregation device 2 is a composite data set ^D j (yi) If so, the update unit 1j3 updates the category y i Data set D j (yi) Update with one of the following: In equation (a), the data set D held by the user device 1j is j (yi) and the composite data set ^D received from aggregator 2. j (yi) The union of these two sets is the updated data set D j (yi) In equation (b), the composite data set ~D obtained by the output data acquisition unit 1j1 is j (yi) and the composite data set ^D received from aggregator 2. j (yi) The union of these two sets is the updated data set D j(yi) It is said that.

[0043] If there is no output data received from the aggregation device 2, the update unit 1j3 of the user device 1j does not update the input data.

[0044] The plurality of user devices 11, ..., 1J and the aggregation device 2 repeat the processing of the output data acquisition unit 1j1, the first transmission unit 1j2, the determination unit 21, the second transmission unit 22, and the update unit 1j3 using the updated input data. Note that, if there is a user device 1j among the user devices 1j for which the input data has not been updated, that user device 1j performs these processing using the input data that has not been updated.

[0045] For example, the update unit 1j3 of the user device 1j updates the input data until a predetermined condition is met. An example of the predetermined condition is that the number of updates of the input data reaches a predetermined number. The predetermined condition may be that the size of the updated input data is equal to or larger than a predetermined size. When the input data is a data set D j (yi) If so, the size of the input data is the data set D j (yi) The number of updates and the predetermined size of the input data may be different for each user device, or may be the same for all user devices.

[0046] The update unit 1j3 of the user device 1j determines whether a predetermined condition is met after updating the input data, for example. If it is determined that the predetermined condition is met, the user device 1j then terminates the above process. In this case, other user devices for which it is not determined that the predetermined condition is met may continue the above process.

[0047] When it is determined that the predetermined conditions are satisfied and the above process is terminated, the user device 1j uses the input data updated at the end of the above repeated process as the composite data. In other words, the input data updated at the end of the above repeated process is used as the final composite data.

[0048] In this way, even if the amount of data held or available by each user device is small, if there are multiple LLM user devices, it is generally expected that the amount of data will be sufficiently large when viewed as a whole. As in the above example, by effectively utilizing the large amount of data held in a distributed manner and by linking with each other via the aggregation device 2, multiple LLM user devices can generate high-quality text using their own economical machine learning systems while protecting their privacy.

[0049] [Modifications] The specific configurations of the embodiments of the disclosed technology are not limited to the configurations described above. The specific configurations of the embodiments of the disclosed technology can be appropriately modified in design, etc., within the scope of the spirit of the embodiments of the disclosed technology.

[0050] For example, the decision unit 21 of the aggregation device 2 may decide to send each of the multiple output data received from multiple user devices 11, ..., 1J to the user device that sent the output data that has a low similarity to the respective output data.

[0051] For example, the output data received from the plurality of user devices 11, . . . , 1J is a composite data set ∼D1 (yi) ,…,~D J (yi) and the composite data set ∼D received from the user device 1j j (yi) The synthetic data set with the lowest similarity to j' (yi) If so, the determination unit 21 determines the composite data set ^D to be transmitted to the user device 1j. j (yi) ~D j' (yi) Let's say.

[0052] In this example, by transmitting to the user device 1j the output data that is most different from the received output data, the size of the input data for the user device 1j is increased most efficiently, thereby reducing the number of iterations required and shortening the processing time.

[0053] In addition, when the output data is composed of multiple data, the decision unit 21 of the aggregation device 2 may decide to which of the multiple user devices 11, ..., 1J the output data selected from the union of the multiple data that constitute each of the output data received from the multiple user devices 11, ..., 1J should be sent.

[0054] For example, the output data received from the plurality of user devices 11, . . . , 1J is a composite data set ∼D1 (yi) ,…,~D J (yi) If so, the determination unit 21 determines the composite data set to be transmitted to the user device 1j as the composite data set ~D1 (yi) ,…,~D J (yi) may be selected from the union of

[0055] For example, ~D j (yi) ={~z1,~z2,…,~z ni (j)}, then the synthetic data set ~D1 (yi) ,…,~D J (yi) The number of elements in the union of is n i (1) +…+n i (J) The determination unit 21 determines this n i (1) +…+n i (J) N elements selected from j It is decided to transmit a composite data set consisting of N elements to the user device 1j. j is 1 or more n i (1) +…+n i (J) is a predetermined integer below.

[0056] N j The selection of elements is j This may be performed so that the similarity between the composite data set composed of these elements and the composite data set transmitted by the user device 1j to the aggregation device 2 becomes low.

[0057] In addition, the synthetic data set ~D1(yi) ,…,~D J (yi) The composite data set may be selected such that the same elements in the union of are transmitted to different user devices.

[0058] The various processes described in the embodiments of the disclosed technology may not only be performed chronologically in the order described, but may also be performed in parallel or individually depending on the processing capacity of the device performing the processes or as needed.

[0059] For example, data may be exchanged directly between the components of the synthetic data generating device, or may be exchanged via a storage unit (not shown).

[0060] Furthermore, a device (terminal) for using the device, system, or method of the present invention via a network (telecommunications line) may also be provided. The "device (terminal) for use" may be provided with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device, system, or method of the present invention.

[0061] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

[0062] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0063] [Program, Recording Medium] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0064] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0065] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0066] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 4, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0067] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0068] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0069] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0070] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. A synthetic data generation system including a plurality of user devices and an aggregation device, wherein each of the plurality of user devices includes an output data acquisition unit that obtains output data by inputting input data into a large-scale language model that the user devices have, and a first transmission unit that transmits the output data to the aggregation device, wherein the aggregation device includes a determination unit that determines which output data, from among the output data received from the plurality of user devices, to send to which of the plurality of user devices, and a second transmission unit that transmits the determined output data to the determined user device, wherein each of the plurality of user devices further includes an update unit that, if output data is received from the aggregation device, updates the input data using the output data received from the aggregation device, wherein the plurality of user devices and the aggregation device repeat the processing of the output data acquisition unit, the first transmission unit, the determination unit, the second transmission unit, and the update unit using the updated input data instead of the input data, and the updated input data at the end of the repeated processing is used as synthetic data.

2. A synthetic data generation system according to claim 1, wherein the determination unit randomly determines which output data from the output data received from the plurality of user devices is to be sent to which of the plurality of user devices.

3. A synthetic data generation system according to claim 1, wherein the decision unit decides to transmit each of the plurality of output data received from the plurality of user devices to a user device that transmitted output data that has a low similarity to the respective output data.

4. A user device comprising: an output data acquisition unit that obtains output data by inputting input data into a large-scale language model that the device has; a first transmission unit that transmits the output data to an aggregation device; a determination unit that determines which output data from the output data received from the plurality of user devices to send to which of the plurality of user devices; and a second transmission unit that transmits the determined output data to the determined user device; and an update unit that updates the input data using the output data received from the aggregation device when output data is received from the aggregation device, wherein the user device and the aggregation device repeat the processing of the output data acquisition unit, the first transmission unit, the determination unit, the second transmission unit and the update unit using the updated input data instead of the input data, and the updated input data at the end of the repeated processing is used as synthetic data.

5. A synthetic data generation method comprising: an output data acquisition step in which an output data acquisition unit obtains output data by inputting input data into a large-scale language model that the output data acquisition unit has; a first transmission step in which a first transmission unit transmits the output data to an aggregation device; and an update step in which, if output data is received from an aggregation device including a determination unit that determines which output data from the output data received from the plurality of user devices to transmit to which of the plurality of user devices, and a second transmission unit that transmits the determined output data to the determined user device, an update step in which an update unit updates the input data using the output data received from the aggregation device; wherein the processing of the output data acquisition step, the first transmission step and the update step is repeated using the updated input data instead of the input data, and the updated input data at the end of the repeated processing is used as synthetic data.

6. A program for causing a computer to execute each step of the synthetic data generation method of claim 5.

Images