Information processing device, information processing system, and information processing method

Abstract

[Problem] To set an appropriate degree of contribution according to provided data. [Solution] This information processing device is provided with one or more memories and one or more processors. The one or more processors acquire a question for which an answer is generated by a first model, acquires information, which is used by the first model in generation of an answer to the question, from a data group on the basis of the question, and calculates a degree of contribution of the information to generation of the answer by the first model.

Description

Information processing device, information processing system, and information processing method

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method.

[0002] Large language models (LLMs), which are language models trained using machine learning, are beginning to be used in a variety of applications. One application of LLMs is Retrieval Augmented Generation (RAG), which searches for information in response to input and uses the searched information to generate output from the LLM.

[0003] S. Borgeaud, et.al., “Improving language models by retrieving from trillions of tokens”, arXiv.org:2112.04426v3, 7 February 2022, https: / / arxiv.org / pdf / 2112.04426v3.pdf

[0004] One non-limiting problem that embodiments of the present disclosure attempt to solve is evaluating information used to generate output by foundational models, such as large-scale language models, or generative models.

[0005] According to one embodiment, an information processing device includes one or more memories and one or more processors, which acquire a question for which an answer is generated by a first model, acquire information used by the first model to generate the answer to the question from a data set based on the question, and calculate a contribution of the information to generation of the answer by the first model.

[0006] According to one embodiment, an information processing system includes one or more information processing devices. The one or more information processing devices acquire first data corresponding to a question for which an answer is generated by a first model, acquire information used by the first model to generate an answer to the question first data from a data group based on the question first data, and calculate a contribution of the information to generation of the answer by the first model. The one or more information processing devices that generate the answer to the question first data and the one or more information processing devices that calculate the contribution of the information are different information processing devices, or at least a portion of the information processing devices are the same information processing device.

[0007] A diagram showing an overview of processing according to an embodiment. A diagram showing an example of contribution calculation processing according to an embodiment. A diagram showing pseudocode of an example of contribution calculation processing according to an embodiment. A diagram showing an example of information management according to an embodiment. A diagram showing an implementation example of an information processing device that constitutes at least a part of an information processing system according to an embodiment.

[0008] The problems to be solved by the embodiments of the present disclosure are not limited to the problems described above, and as further examples of some problems that are not limited to these, problems corresponding to the effects described in the embodiments can also be considered. In other words, a problem corresponding to at least one of the effects described in the description of the embodiments of the present disclosure can be considered to be a problem to be solved by the present disclosure.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings and the description of the embodiments are given by way of example only and are not intended to limit the present invention.

[0010] For example, in this disclosure, a case will be described in which text data such as characters and sentences, which are information for a language model, is used, but the embodiment of the present invention is not limited to this and can also be applied to data such as images, audio, video, sensor data, etc. By appropriately tokenizing these data, they can be similarly applied in the forms described below.

[0011] Large-scale language models that use search results require a large amount of data related to the information being searched. Furthermore, this data must cover a wide range of areas in the field being searched, and the extent to which the data is used as search results, i.e., how useful the data is, is also important.

[0012] In order to obtain highly useful data, it is possible to assign contribution levels to providers of data used in searches according to the usefulness of the data, and to provide incentives according to these contribution levels. By setting appropriate contribution levels, it is possible to improve the overall quality of search data.

[0013] 1 is a diagram illustrating an outline of a process according to an embodiment. The process in this diagram may be executed, for example, in an information processing system including one or more processors installed in one or more computers and one or more storage devices (including those provided in the computers). The transmission and reception of each piece of data and the storage location of each piece of data can be set arbitrarily as long as no inconsistencies occur depending on the system configuration.

[0014] In the following, the processing of at least one processor in the information processing system is described, but even if there is no particular limitation to at least one processor, it may refer to the execution of processing by at least one processor, or it may refer to the processing flow of the information processing system as a whole. These may be interpreted appropriately within the scope of the technical idea of ​​the present disclosure.

[0015] The information processing system is a system that generates and outputs third data, which is output data for first data, which is input data, using a first model, which is a probabilistic model trained using pre-learning data. The first model may be, for example, a base model or a generative model.

[0016] The first data is, for example, text for asking a question to the first model, and is called a prompt. Depending on the context, the first data can be interpreted as a question for which an answer is generated by the first model. In this disclosure, the term "question" may conceptually include either a question from a user or an instruction from a user. Also, in this disclosure, the term "question" may conceptually include either data input by a user or processed data obtained by performing predetermined processing on that data.

[0017] As described above, the first data may include any of images, audio, video, and sensor data. The third data is, for example, text indicating an answer to a question. In the present disclosure, the "answer" may conceptually include either an answer to a question or a response to an instruction. Furthermore, in the present disclosure, the "answer" may conceptually include either data generated by the first model or processed data obtained by performing predetermined processing on that data. Like the first data, the third data may include any of images, audio, video, and sensor data. Furthermore, both the first data and the third data are not limited to these data and may be data in other formats.

[0018] The first model is stored in a storage device within the information processing system and is formed by a processor within the information processing system referencing the storage device. The information processing system performs inference using the first model formed by the processor.

[0019] The information processing system first receives first data as input data (S10). The input of data may be performed to at least one processor in the information processing system via various interfaces. For example, the at least one processor in the information processing system may receive the first data input by a user as input data.

[0020] The first data input by the user is, for example, data input or specified by the user via the user interface of the information terminal, and may be text entered using a keyboard, text selected in a sentence, an image dragged and dropped with a mouse, or audio picked up by a microphone, etc. Here, the user's information terminal is an example of an information processing device and is one element constituting an information processing system.

[0021] At least one processor in the information processing system acquires second data for performing a search necessary to generate third data based on the first data received as input data (S12). The second data is, for example, a query to be input to a search engine. Information searched using this second data is used as additional input when determining the output of the third data. In other words, the second data is data for searching for information related to an answer to the first data.

[0022] At least one processor in the information processing system may, for example, tokenize the first data by inputting the first data into a trained first model, and acquire a query required for a search as the second data based on the acquired tokens. As a non-limiting example, the first model may be a large-scale language model, and in this case, the processor acquires the second data by extracting a search query for generating third data from the tokenized first data for the input first data indicating a question or the like.

[0023] Furthermore, at least one processor in the information processing system may, for example, vectorize the first data and acquire the acquired vector as second data, which is a query required for a search. Furthermore, at least one processor in the information processing system may, for example, acquire the first data itself as second data, which is a query required for a search. Furthermore, at least one processor in the information processing system may use a conventional method to acquire second data, which is a query for a search engine described below, based on the first data.

[0024] At least one processor in the information processing system uses the acquired second data as a query to search for information required to generate the third data using a search engine (S14). The search engine is stored in a storage device in the information processing system, similar to the first model, and is formed by at least one processor in the information processing system referencing this storage device. The information processing system performs a search for the required information using the search engine formed by the processor.

[0025] The information processing system inputs the second data into a search engine and acquires information to be used to generate the third data from the data group (S16). The processing of the search engine may be executed by at least one processor in the information processing system. As a non-limiting example, this processing enables the information processing system to acquire information to be used to generate an answer to the first data from the data group based on the first data.

[0026] The processing by the search engine can be performed by, for example, using the data to be searched, which is information within a data group, as a key to extract a key for the query. Note that the search for the key by the search engine is not limited to a specific method, and any general method for obtaining a key for a query can be used.

[0027] The key and the query may be encoded. For example, the search engine may execute a search using a function defined by the encoded key and the encoded query. The key and query encoding may also be pre-trained during pre-training of the first model.

[0028] As a non-limiting example, if the first model is a large-scale language model, at least one processor in the information processing system can generate a query required for the search from the tokenized text and use this query to reference keys in the data group to extract information that may include text required to answer the input text.

[0029] Furthermore, the search engine may be an engine formed by any method, such as a model formed by keyword matching, an attention mechanism, or a model formed by a learning method using reinforcement learning, etc. However, it is not limited to these, and an engine formed by other methods may also be used.

[0030] At least one processor in the information processing system reflects the search results of the search engine in generating output data in the first model (S18).

[0031] At least one processor in the information processing system generates third data based on the input data received in S10 and the information obtained by the search to generate third data, which is output data in the first model (S20). That is, the at least one processor in the information processing system can use this information to generate an answer to the first data using the first model.

[0032] At least one processor in the information processing system may output the generated third data (answer) to the information terminal of the user who input the first data in S10. The output third data (answer) may be, for example, displayed on the user's information terminal, read aloud on the information terminal, printed via the information terminal, etc.

[0033] For example, at least one processor in the information processing system can input the input data and the information obtained by the search into the first model to generate a response or answer using this information in addition to knowledge based on the pre-trained data as third data. As an example, at least one processor in the information processing system can process the input data and the information obtained by the search into a prompt in a predetermined format and input this prompt into the first model to generate the third data.

[0034] As another non-limiting example, at least one processor in the information processing system may input information obtained by the search into the first model as input to the first model's internal calculations, and input the input data as a prompt to the first model to generate third data. In this example, the information input point is appropriately designed depending on the architecture of the first model, and may be, for example, any of the input layer, intermediate layer, and output layer of the first model.

[0035] In this disclosure, inputting or incorporating information obtained by a search into a first model, or outputting third data from such a first model, may be referred to as augmenting the first model with search results (information obtained by a search) or augmenting the output of the first model with a search.

[0036] The third data is generated, for example, as data obtained by augmenting the output of the first model for the first data through a search. The third data generated in this manner can be data that reflects more recent information than when the first model is simply used.

[0037] The first model can generate the third data using information about the encoded key set in the data group. As a non-limiting example, when the first model is a large-scale language model, the first model obtains information as a search result for input text as an encoded embedding vector, and reflects the information of this encoded embedding vector in output data, thereby generating more accurate text, etc. as an answer to a question, etc.

[0038] At least one processor in the information processing system calculates and assigns a contribution level for information obtained from the data group and used to generate the third data (S22). The contribution level is an example of an index for evaluating the usefulness of information. The contribution level may be assigned at the time the information is provided to the data group. By this process, the information processing system can, as a non-limiting example, calculate the contribution level of the information in generating an answer using the first model.

[0039] The above is the overall processing flow of the information processing system including the contribution degree calculation process according to the present disclosure. Next, calculation of the contribution degree for each piece of information in a data group in the information processing system will be described in detail. The information processing system can calculate the contribution degree of the information when the information is provided to the data group to be searched or when the third data is generated.

[0040] 2 is a flowchart showing an example of a first-stage process of calculating a contribution degree. As the first-stage contribution degree assignment, the information processing system can set a contribution degree to the information at the timing when the first data is input and the third data is generated.

[0041] However, the information processing system may calculate the first-stage contribution degree at a later timing. As another example, the information processing system may calculate the first-stage contribution degree at the timing when it is determined that the data searched by the search engine in the first model will be used to generate the third data.

[0042] The first stage of assigning contribution points involves obtaining an evaluation that indicates the degree of influence that the information provided to the data group has on the distribution of third data that is generated by searching the data group.

[0043] In other words, when generating third data through augmentation with search, the contribution of this first stage can be used as a necessary indicator of the extent to which the information provided can influence the overall data generated.

[0044] When a user attempts to obtain third data using the first model and the search engine, at least one processor in the information processing system can set a contribution to the provided information.

[0045] At least one processor in the information processing system accepts information provided by a data provider via an appropriate interface (S100), such as, but not limited to, news articles, internal company documents including personnel and technical information, health information, image datasets, video datasets, experimental datasets, genomic data, and confidential information in general.

[0046] As with the input / output data, the information provided by the provider is not limited to text, but may include at least image, audio, video, sensor data, and other data.

[0047] The provider of this information may be, for example, a user related to the field of the data group, or in the case of a news article, the source of the news, or in the case of an internal document, the company or department that provided it.

[0048] Upon specification by a user who is a provider, the information processing system can also accept data provision by automatically crawling information uploaded to a specified database, storage device, or website located in a domain. In this case, in the case of information provided by other providers on a curation site, etc., the provider of such information can be the source of the information referenced by the curation site, and the curation site, etc. can be excluded from being considered as a provider.

[0049] The provided information may be newly stored in a storage area within the information processing system, or may be newly stored in a storage area outside the information processing system. Furthermore, the information processing system may refer to information in a storage area outside the information processing system where information is already stored (for example, a location specified by the user).

[0050] At least one processor in the information processing system can store the information provided by the provider in any location. For example, the information may be stored in a storage area that requires security. In this case, the provider can grant the information processing system permission to access the information.

[0051] At least one processor in the information processing system may link and manage the received information with various related information, such as the management data presented in an example, not limited to case (A) or case (B), in Figure 4.

[0052] At least one processor in the information processing system can link and manage the information ID, provider ID, number of uses, and total royalty amount for each piece of provided information, as illustrated in (A), by registering in one record in the database information for uniquely identifying the information (information ID), information for uniquely identifying the provider of the information (provider ID), the number of times the information is used in generating an answer using the first model, and the total amount of royalties to be granted to the provider in accordance with the use of the information.

[0053] Furthermore, at least one processor in the information processing system can link and manage the date and time of use, contribution, royalty amount, question, and answer by registering, for example, as illustrated in (B), the date and time when the information was used in generating an answer using the first model (date and time of use), the degree of contribution of the information in generating an answer using the first model, the amount of royalty according to that degree of contribution, the question (first data) that triggered the use of the information, and the answer (third data) generated by the first model using that information, in one record in the database.

[0054] At least one processor in the information processing system may acquire, for each piece of information, all or part of the related information related to that information by referring to the management data. At least one processor in the information processing system may provide all or part of the related information related to that information to the information provider or the provider's information processing terminal, for example, in response to a request from the information provider or the provider's information processing device.

[0055] At least one processor in the information processing system receives first data, which is input data from a user using the first model, and generates third data for the first data (S102). The at least one processor in the information processing system generates second data from the first data input by the user, and executes a search based on the second data using a search engine.

[0056] At least one processor in the information processing system calculates the contribution in the first stage based on the distribution of third data generated reflecting search results obtained by performing a search using the provided information as search target data, and the distribution of fourth data, which is data generated without performing a search (S104). In other words, the at least one processor in the information processing system can calculate the contribution using the first model reinforced with the search results and the first model not reinforced with the search results.

[0057] At least one processor in the information processing system can calculate the contribution by, for example, comparing the first model augmented with the search results with the first model not augmented with the search results, where the fourth data does not need to be actually generated.

[0058] At least one processor in the information processing system calculates the first-stage contribution using, for example, the distance between the distribution of the third data and the distribution of the fourth data or a quantity equivalent to the distance. As a non-limiting example, the information processing system may calculate the contribution using the KL divergence (Kullback-Leibler Divergence) between the distribution of the third data and the distribution of the fourth data as a quantity equivalent to the distance. Of course, the information processing system can calculate the contribution using other distances or statistics.

[0059] The KL divergence is expressed by formula (1), where P is the distribution of the third data generated by performing a search using a data set including the provided information, and P' is the distribution of the fourth data generated without performing a search. The information processing system can obtain the KL divergence by using sampled data based on a probability distribution.

[0060] At least one processor in the information processing system performs a search using a set of data after information is provided in response to one prompt (first data), and obtains a sample S of third data according to a probability distribution P. The at least one processor in the information processing system repeats this process for the one prompt (first data) multiple times, for example, N times, and obtains samples S1, S2, ..., S according to the distribution P. N Get.

[0061] At least one processor in the information processing system calculates the probability P(S) of each sample in the third data distribution. i ) and the probability of each sample in the distribution of the fourth data, P'(S i ) and based on this, the KL divergence can be obtained as the first-stage contribution that indicates how much the provided data contributes.

[0062] At least one processor in the information processing system calculates the KL divergence by expressing the sampling result in a discretized expression such as equation (2). That is, the information processing system may calculate a value that approximates the KL divergence. Note that the value that approximates the KL divergence may be calculated using an equation other than equation (2). Sampling may be performed, for example, according to Monte Carlo sampling.

[0063] FIG. 3 is a pseudocode showing an example of a process for obtaining the KL divergence based on this equation (2).

[0064] At least one processor in the information processing system initializes a variable sum indicating the KL divergence amount.

[0065] At least one processor in the information processing system extracts N samples from the distribution. The information processing system, for example, obtains the N samples by repeating the generation of third data (sample generation) for one prompt N times as described above. As another non-limiting example, instead of generating samples N times, the information processing system may generate a number of third data pieces that is sufficiently greater than N and randomly extract N samples from the generated third data pieces.

[0066] At least one processor in the information processing system calculates, for the extracted sample, a probability P(S i ), P'(S i ) are obtained for each of the extracted N samples. The difference d between the logarithms of the respective probabilities is calculated for each of the extracted N samples, and the sum of these differences d across the N samples is calculated to calculate the contribution of the first stage. The information processing system may also normalize the sum by dividing it by N.

[0067] Such KL divergence can be used as an index indicating whether an output using data not included in the pre-training data used in training the first model can be provided via the first model as an output appropriately interpolated with newly provided information. In other words, the larger the value of this KL divergence, the more appropriately the information provider can provide data not included in the pre-training data.

[0068] Conversely, if the difference between the distribution (distribution shape) of the third data and the distribution (distribution shape) of the fourth data is small and the difference in KL information is small, it indicates that the provided information is likely to have already been used in training. In other words, even if a search is performed using the provided data, the same answer can be obtained as if the already trained first model were run without the search process.

[0069] By setting the total number of samples N to a sufficiently large predetermined value, the KL divergence can be obtained as an approximation of equation (2). At least one information processing device in the information processing system may use this discretized KL divergence as the contribution of the first stage, or may calculate the contribution of the first stage by performing some kind of calculation, such as weighting, on the obtained KL divergence.

[0070] At least one processor in the information processing system may reflect the first-stage contribution to the provider who provided the information, and may associate and store the provider (provider ID) and the information (information ID) directly in one record or indirectly in multiple records, as illustrated in Figure 4 (S106). Also, at least one processor in the information processing system may increment the number of uses of the information obtained by the search in management data such as that shown in Figure 4 (A). The contribution to the provider may be accumulated, for example, by adding up the contributions to the information provided up to that point, and may be used as the total first-stage contribution to the provider.

[0071] Furthermore, at least one processor in the information processing system can weight information so that older information is weighted lightly, or can delete information that has been around for a certain amount of time from the total contribution score. By processing information in chronological order in this way, it is possible to provide information that takes older information into consideration while weighting newer information.

[0072] As described above, when new information to be searched is provided to a data group, the information processing system according to the above embodiment can set an appropriate contribution level for the provided data. Furthermore, it can also process the sum of contribution levels for each provider. The information processing system can use the first-stage contribution level as a necessary condition for determining whether the provided data can affect the output for the data provider.

[0073] In the above, the probability distribution P of the third data is used as a starting point, but the present invention is not limited to this. For example, starting from the probability distribution P' of the fourth data, a sample P'(S i ') and get this S i The third data for ' is sampled from the probability distribution P(S i The contribution may be calculated using the KL divergence KL(P'||P) obtained by

[0074] Furthermore, the provided information that forms the probability distribution P with a high KL divergence may be used as training data for re-training the first model. By reflecting such data as training data, it is possible to improve the accuracy of the output data in the first model.

[0075] Furthermore, the information processing system may repeatedly perform the process of FIG. 3 multiple times and calculate the degree of contribution based on the number of times the KL divergence becomes higher than a predetermined value.

[0076] The calculation of the contribution in the first stage is not limited to the above example. Some further examples are given below, but the calculation is not limited to these, and other methods that can reflect the complementarity of information when a search is performed appropriately may also be used.

[0077] The information processing system can input a prompt (first data) into a first model augmented with search results to generate data (e.g., text), calculate the likelihood that this data will be output from the first model before augmentation (when search results are not used), and calculate the first-stage contribution based on this likelihood. The information processing system can, for example, use the inverse of this likelihood as the first-stage contribution.

[0078] The information processing system may generate data by inputting the prompt (first data) into a first model augmented with search results and a first model not augmented with search results, and may input these two pieces of data into a model (e.g., a model trained by machine learning) that encodes data into latent variables, and calculate the first-stage contribution based on each latent variable. The information processing system may use, for example, an index such as the Euclidean distance between two latent variables in a latent space of latent variables, or the value obtained by subtracting the cosine similarity between the two latent variables from 1, as the first-stage contribution.

[0079] The information processing system can calculate the contribution of the first stage based on the cross-entropy of the distribution P' of the fourth data relative to the distribution P of the third data when the prompt (first data) is input. The information processing system may use this cross-entropy as the contribution of the first stage, or may use a normalized amount of the cross-entropy as the contribution of the first stage, for example.

[0080] Next, we will explain the process of calculating the second-stage contribution level based on whether the provided information was useful to the answer, in addition to the influence that the provided information had on the answer. The information processing system may calculate the second-stage contribution level at the timing when the information is provided (at S104 in FIG. 2), as in the calculation of the first-stage contribution level described above, or as another example, at the timing when third data is actually generated using the first model and a search engine in response to a user request (at S22 in FIG. 1).

[0081] At least one processor in the information processing system can calculate a second-stage contribution by evaluating whether the provided data actually contributed usefully to the result. This second-stage evaluation can be realized, for example, using a second model different from the first model. The information processing system can use, as an evaluator, for example, the second model, which is an evaluation model trained for evaluation, and which, when inputted with the input (first data) and output (third data or fourth data) of the first model, returns a higher evaluation value the more correct the result (the higher the correctness, validity, or validity).

[0082] For example, if the input of the first model is a question from a user and the output of the first model is an answer to the question, the second model, which is an evaluator, may evaluate whether the output is an answer to the question, i.e., the legitimacy, validity, or appropriateness of the output as an answer to the question. At least one of these evaluations may include, as an evaluation item, the correctness of the content of the output as an answer or the extent to which the output maintains the format (style) of an answer. In the case of natural language processing, the evaluator may evaluate each evaluation item by performing semantic analysis or contextual analysis.

[0083] Additionally, evaluating the legitimacy, validity, or appropriateness of a generated answer conceptually includes evaluating the generated answer itself, as well as evaluating the answer by comparing the output from the first model that generated the answer, augmented with search results (third data), with the output from the first model that did not augment with search results (fourth data).

[0084] At least one processor in the information processing system can, for example, use the second model to calculate the contribution as an expected value using a probability distribution P of third data generated using the provided data as search data and a probability distribution P' of fourth data generated without performing a search, as shown below.

[0085] Here, C(S) is the output value of the evaluator for sample S, and E(·) denotes the expected value. The information processing system calculates C using the expected value of C for sample S obtained according to probability distribution P and the expected value of C for sample S' obtained according to probability distribution P'. More specifically, one or more samples are taken from the random variables S to P(S) and S' to P'(S'), and the expected values ​​of C(S) and C(S') are calculated using the average of these samples.

[0086] In this way, the information processing system can calculate the second-stage contribution by sampling based on the probability distributions P and P' and calculating the expected value of the evaluation value. In other words, the information processing system can calculate the contribution using a first model reinforced with search results, a first model not reinforced with search results, and a second model that evaluates answers generated by the first model. The second model that outputs the evaluation value can be trained using any method. As described above, the more correct the answer, the higher the evaluation value calculated by this second model.

[0087] Therefore, according to formula (3), the information processing system can calculate the difference between the expected value of the evaluation in the probability distribution of the third data generated by using the provided information for a search and the expected value of the evaluation in the probability distribution of the fourth data generated without performing a search as the contribution of the second stage.

[0088] In other words, when the provided information is used in a search, the more correct the answer, the greater the contribution value in the second stage.

[0089] As described above, the information processing system of the above form can calculate a contribution that reflects the degree of legitimacy, validity, and appropriateness of the information provided in the data group in the answer, and how effective it is in obtaining a correct answer.

[0090] The information processing system can use both the first-stage contribution degree and the second-stage contribution degree, or can use only one of the contribution degrees. By setting the contribution degrees as described above, the information processing system can encourage information providers to provide information that improves the coverage rate of information in the first-stage contribution degree, and to encourage information providers to provide information that improves the legitimacy, validity, and appropriateness of the data answered in the second-stage contribution degree.

[0091] The degree of contribution calculated above can be used, as a non-limiting example, as an index for realizing some kind of processing, such as a payback, as royalty to the information provider. The information processing system may, for example, award royalties (money or equivalent compensation) to the information provider in an amount corresponding to the degree of contribution of the information. Royalty is a concept that includes incentives, rewards, usage fees, points, etc. Awarding royalties to the information provider may be done, for example, by linking the amount of royalty to the information provider's identification code, or by other methods.

[0092] In the above, when the information processing system awards royalties to a provider of information that is obtained from a data set by a search and used to generate an answer using the first model, the amount of the royalties is determined according to the degree of contribution of the information. The method of awarding royalties is not limited to the above example.

[0093] For example, the information processing system may not calculate the contribution of information obtained from a data set through a search and used to generate an answer using a first model, but may instead award a flat amount of royalty to the provider of the information regardless of the degree of contribution.

[0094] Alternatively, the information processing system may, for example, calculate the contribution of information obtained from a data set by a search and used to generate an answer using the first model, and if the contribution exceeds a certain level, grant a flat-rate royalty to the provider of the information.

[0095] That is, the information processing system may be configured to obtain a question (first data) for which an answer is generated by a first model, and based on that question, to obtain information to be used to generate an answer to the question (third data) from a data set by searching, and to grant royalties to the provider of that information.

[0096] The information processing system can be broadly divided into an answer generation system using the first model, a search system using a search engine, and a data set used by the search engine. The information processing system can provide any of these as an API (Application Programming Interface).

[0097] For example, it is possible to provide a system that provides an API for input and output of the first model as a whole, and obtains answers with search reinforcement using information provided by an information provider as a search target.

[0098] For example, it is possible to provide a system that provides an API related to input and output of a search engine and uses information provided by an information provider as a search target.

[0099] For example, an API for accessing a data set can be provided, and a route for accessing data used in a search engine search can be provided as a system.

[0100] In this way, the division of the system can be arbitrarily performed. Regardless of the provision of any of the above APIs, the information processing system according to the above embodiment makes it possible to give appropriate contribution points to data providers.

[0101] That is, the information processing system may be in a form including at least one of the first model, the search engine, and at least a part of the sub-data group in the data group.

[0102] Furthermore, the information processing system may include, as multiple different information processing devices, an information processing device that generates the third data and an information processing device that calculates the degree of contribution, or may be configured such that the generation of the third data and the calculation of the degree of contribution can be realized in the same information processing device. That is, the information processing system includes one or more information processing devices, and the processing of the present disclosure in these information processing devices can be appropriately distributed or aggregated and executed among the one or more information processing devices.

[0103] When an information processing system has a first model, the information processing device that sets the contribution degree can calculate the above-mentioned contribution degree from third data that is output from the first model that uses data obtained by passing second data output from the first model held by the system to a search engine inside or outside the information processing system, and fourth data that is output from the first model without executing a search. An ID is assigned to each piece of data (information) in a data group, and a contribution degree is set for this ID (information ID). It is also possible to obtain data provided by the same provider from the ID and summarize their contribution degrees.

[0104] If the information processing system has a search engine, the information processing system that sets the contribution level can acquire third data and fourth data based on the output from the first model inside or outside the information processing system and set the contribution level. When the third data is generated, the results obtained using the search engine within the information processing system can be passed to the first model.

[0105] When an information processing system has at least a portion of a data group, the information processing system that sets the contribution level can obtain the third data and the fourth data using a first model and a search engine inside or outside the information processing system at the time the information is provided, set the contribution level, and assign it to each piece of provided information in the data group.

[0106] The data set does not need to be stored together, and some of the data may be stored in secret, and an API may be provided to access the encrypted key. For example, confidential data of multiple companies may be stored in secret at each company, and paths and interfaces for accessing the data may be prepared.

[0107] In this case, the sub-data sets that exist for each company can be hidden from the outside, so a user who wants an answer using the first model cannot detect which company's data the search is based on. On the other hand, the information processing system can determine which sub-data set the answer is based on by referencing the ID (information ID) linked to the data in the data set.

[0108] In addition, the information processing system may output information indicating the provider of the information (information obtained by search) used to generate the answer using the first model (e.g., a provider ID, a string indicating the provider's name, and / or a string indicating the source or source of the information, such as a URL, etc.) in a manner that allows the user receiving the answer to identify that the answer was generated by reference to information provided by that provider.

[0109] Furthermore, the information processing system may output information indicating the provider of the information (information obtained by search) used to generate the answer using the first model (e.g., a provider ID, a character string indicating the provider's name, and / or a character string indicating the source or location of the information, such as a URL) in a manner (e.g., outputting it together with the answer) that enables a user who receives the answer (a recipient of the answer) to identify that the answer was generated by referring to (using) the information provided by the provider. The information indicating the provider may be text, an image, or the like.

[0110] The information processing system can also assign a contribution level to information providers who have source sub-data groups identified in this way during actual operation, based on the number of times and frequency that the data is used as an answer.

[0111] In either case, the first model may have various forms. As described above, the large-scale language model may be a model that inputs text such as sentences or words and outputs text such as sentences or words. As another example, the first model may be a model that inputs text and outputs an image or video, a model that inputs an image or video and outputs text, a model that inputs an image or video and inputs an image or video, a model that inputs graph information and outputs graph information, a model that inputs graph information and outputs text, an image, or video, a model that inputs text, an image, or video and outputs graph information, etc. Of course, the first model may be a model other than these.

[0112] The trained model in the above-described embodiment may be a concept that includes, for example, a model that has been trained as described and then further distilled using a general method.

[0113] In the following, each device in the embodiments may refer to at least one device present in an information processing system. The device may be, for example, a device that sets a contribution level when information is provided to a data group in the information processing system, or a device that sets a contribution level when third data is generated, or a device that uses a first model, or a device that uses a search engine, or a device that performs at least two or more of these operations in combination.

[0114] Some or all of the devices (information processing devices provided in the information processing system) in the above-described embodiments may be configured as hardware, or may be configured as software (programs) executing information processing by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), etc. When configured as software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communications network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), thereby allowing the software information processing to be executed by hardware.

[0115] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk or memory. The storage medium may be located inside the computer (such as a main memory or auxiliary memory), or may be located outside the computer.

[0116] 5 is a block diagram showing an example of the hardware configuration of each device (information processing device provided in the information processing system) in the above-mentioned embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, all of which are connected via a bus 76.

[0117] Although the computer 7 in FIG. 5 includes one of each component, it may include multiple of the same component. Also, while FIG. 5 shows one computer 7, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, a distributed computing configuration may be used in which each computer communicates with the other computers via a network interface 74 or the like to execute the processing. In other words, each device (information processing device provided in the information processing system) in the above-described embodiment may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize its functions. Furthermore, the system may be configured so that information sent from a terminal is processed by one or more computers provided on a cloud, and the processing results are sent to the terminal.

[0118] The various calculations of each device (information processing device provided in the information processing system) in the above-described embodiments may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be implemented by at least one processor and storage device provided on a cloud that can communicate with computer 7 via a network. Thus, each device in the above-described embodiments may be implemented in the form of parallel computing using one or more computers.

[0119] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls or performs calculations on a computer. The processor 71 may also be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. The processor 71 may also include an optical circuit or a calculation function based on quantum computing.

[0120] The processor 71 may perform arithmetic processing based on data or software input from each device, etc., configured inside the computer 7, and may output arithmetic results or control signals to each device, etc. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.

[0121] Each device (information processing device provided in the information processing system) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wire or wirelessly.

[0122] The main memory device 72 may store instructions executed by the processor 71, various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data, etc. in each device (information processing device provided in the information processing system) in the above-mentioned embodiments may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by an internal memory built into the processor 71. For example, the memory unit of the information processing device provided in the information processing system in the above-mentioned embodiments may be realized by the main memory device 72 or the auxiliary memory device 73.

[0123] When each device (information processing device provided in the information processing system) in the above-described embodiments is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. At least one storage device may be connected to one processor. A configuration in which at least one processor among multiple processors is connected to at least one storage device among multiple storage devices may also be included. This configuration may also be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may also be included.

[0124] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wire. The network interface 74 may be an appropriate interface, such as one that conforms to an existing communication standard. The network interface 74 may exchange information with an external device 9A connected via the communication network 8. The communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0125] The device interface 75 is an interface such as USB that directly connects to the external device 9B.

[0126] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device directly connected to the computer 7.

[0127] For example, the external device 9A or the external device 9B may be an input device. The input device may be a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device equipped with an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0128] Furthermore, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the external device 9A or the external device 9B may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0129] Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

[0130] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (information processing device provided in the information processing system) in the above-described embodiment. That is, the computer 7 may transmit some or all of the processing results to the external device 9A or the external device 9B, or may receive some or all of the processing results from the external device 9A or the external device 9B.

[0131] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, a-b, a-c, b-c, or a-b-c. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, etc. It also includes the addition of elements other than the listed elements (a, b, and c), such as having d, as in a-b-c-d.

[0132] In this specification (including claims), when expressions such as "using / using data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data, unless otherwise specified. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used as output, unless otherwise specified.

[0133] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0134] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached to it.

[0135] When used in this specification (including the claims), terms implying containing or possessing (e.g., "comprising" or "including" and "having"), they are intended to be open-ended terms that include containing or possessing things other than the object designated by the object of the term. When the object of such terms implies no quantity or a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

[0136] In this specification (including the claims), although expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles a or an) should be interpreted as not necessarily being limited to a specific number.

[0137] In this specification, when a particular advantage / result is described as being obtained from a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained from one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or circumstances, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or circumstances are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0138] When terms such as "maximize" and "maximization" are used in this specification (including the claims), they include finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted accordingly according to the context in which the term is used. They also include finding approximations of these maxima probabilistically or heuristically. Similarly, when terms such as "minimize" and "minimization" are used, they include finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted accordingly according to the context in which the term is used. They also include finding approximations of these minima probabilistically or heuristically. Similarly, when terms such as "optimize" and "optimization" are used, they include finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted accordingly according to the context in which the term is used. It also includes finding approximations of these optimum values ​​probabilistically or heuristically.

[0139] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when expressions such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) are used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may also include an electronic circuit or a device including an electronic circuit.

[0140] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0141] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the content defined in the claims and their equivalents. For example, when numerical values ​​or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure.

Claims

1. An information processing device comprising: one or more memories; and one or more processors, wherein the one or more processors obtain a question for which an answer is generated by a first model which is a generative model; obtain information from a data group based on the question, the information being used by the first model to generate the answer to the question; and grant royalties to a provider of the information used to generate the answer by the first model.

2. The information processing device according to claim 1, wherein the one or more processors manage the number of times the information is used.

3. The information processing device according to claim 1, wherein the one or more processors manage, for the information, a date and time when the information was used to generate the answer.

4. The information processing device according to claim 1, wherein the one or more processors manage, for the information, the question that triggered the use of the information, and the answer generated by the first model using the information.

5. The information processing device according to claim 1, wherein the royalty is an amount according to the contribution of the information in generating the answer by the first model.

6. The information processing device according to claim 1, wherein the royalty is a flat rate.

7. The information processing device according to claim 1, wherein the one or more processors grant the royalty to the information provider according to the contribution of the information to the generation of the answer by the first model.

8. An information processing device as described in any of claims 1 to 7, wherein the one or more processors output information indicating the provider of the information used to generate the answer using the first model in a manner that allows a recipient of the answer to identify that the answer was generated using information provided by that provider.

9. An information processing device as described in any one of claims 1 to 7, wherein the one or more processors calculate the contribution of the information in generating the answer by the first model using the first model augmented with the information and the first model not augmented with the information.

10. An information processing device according to any one of claims 1 to 7, wherein the first model is a probabilistic model.

11. An information processing device according to any one of claims 1 to 7, wherein the first model is a language model.

12. An information processing device according to any one of claims 1 to 7, wherein the one or more processors use the information to generate an answer to the question using the first model.

13. An information processing device as described in any of claims 1 to 7, wherein the one or more processors obtain second data for searching for information to be used by the first model to generate an answer to the question, and obtain the information from the data group by inputting the second data into a search engine and performing a search.

14. The information processing device of claim 13, wherein the second data is data obtained by inputting the question into the first model, and the answer is data generated by inputting information obtained using the search engine based on the second data into the first model.

15. An information processing device as described in any of claims 1 to 7, wherein the one or more processors calculate the contribution of the information so that the contribution of the information in generating the answer by the first model is greater when data generated using the first model augmented with the information is more valid than data generated using the first model not augmented with the information.

16. An information processing device as described in any of claims 1 to 7, wherein the one or more processors calculate the contribution of the information in generating the answer by the first model based at least on a distribution of data generated using the first model augmented with the information and a distribution of data generated using the first model not augmented with the information.

17. An information processing device according to any one of claims 1 to 7, wherein the one or more processors calculate, for each provider of provided information, the contribution of the information in generating the answer by the first model.

18. An information processing device according to any one of claims 1 to 7, wherein at least a portion of the data in the data group is managed in confidentiality by the provider of the data.

19. An information processing device comprising: one or more memories; and one or more processors, wherein the one or more processors obtain a question for which an answer is generated by a first model which is a generative model; cause a search engine to obtain information from a data group based on the question, the information being used by the first model to generate an answer to the question; output an answer to the question based on the information generated using the first model; and grant royalties to a provider of the information used to generate the answer by the first model.

20. An information processing system comprising: one or more memories; and one or more processors, wherein the one or more processors obtain a question for which an answer is generated by a first model which is a generative model; obtain information from a data group based on the question, the information being used by the first model to generate an answer to the question; generate an answer to the question based on the information using the first model; and grant royalties to a provider of the information used to generate the answer by the first model.

21. An information processing method executed by one or more processors, comprising: obtaining a question for which an answer is generated by a first model which is a generative model; obtaining information from a data group based on the question, the information being used by the first model to generate an answer to the question; generating an answer to the question based on the information using the first model; and granting royalties to a provider of the information used to generate the answer by the first model.

Images