Information processing device and information processing system

Abstract

To set an appropriate degree of contribution according to provided data.SOLUTION: An information processing device is provided with one or more memories and one or more processors. The one or more processors acquire a question for which an answer is generated by a first model, acquires information, which is used by the first model in generation of an answer to the question, from a data group on the basis of the question, and calculates a degree of contribution of the information in generating the answer by the first model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing system. [Background technology]

[0002] Large language models (LLMs), which are language models trained by machine learning, are beginning to be used in a variety of applications. One application of large language models is a method called Retrieval Augmented Generation (RAG), which searches for information for an input and uses the searched information to generate an output by the large language model. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] S. Borgeaud, et.al., “Improving language models by retrieving from trillions of tokens”, arXiv.org:2112.04426v3, 7 February 2022, https: / / arxiv.org / pdf / 2112.04426v3.pdf Summary of the Invention [Problem to be solved by the invention]

[0004] One non-limiting problem that embodiments of the present disclosure attempt to solve is evaluating information used to generate output by foundational models, such as large-scale language models, or generative models. [Means for solving the problem]

[0005] According to one embodiment, an information processing device includes one or more memories and one or more processors that acquire a question for which an answer is generated by a first model, acquire information used by the first model to generate an answer to the question from a data set based on the question, and calculate a contribution of the information to generation of the answer by the first model.

[0006] According to one embodiment, an information processing system includes one or more information processing devices. The one or more information processing devices acquire first data corresponding to a question to which an answer is generated by a first model, acquire information used by the first model to generate an answer to the question first data from a data group based on the question first data, and calculate a contribution of the information in generating the answer by the first model. The one or more information processing devices that generate the answer to the question first data and the one or more information processing devices that calculate the contribution of the information are different information processing devices, or at least a part of them are the same information processing device. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an outline of a process according to an embodiment. [Diagram 2] FIG. 11 is a diagram showing an example of a contribution degree calculation process according to an embodiment. [Diagram 3] FIG. 13 is a diagram showing a pseudo code of an example of a contribution calculation process according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of information management according to an embodiment. [Diagram 5] FIG. 1 is a diagram showing an implementation example of an information processing device constituting at least a part of an information processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The problems to be solved by the embodiments of the present disclosure are not limited to the problems described above, and as examples of some further problems that are not limited to the problems, they can be problems corresponding to the effects described in the embodiments. In other words, a problem corresponding to at least one of the effects described in the description of the embodiments of the present disclosure can be the problem to be solved by the present disclosure.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The drawings and the description of the embodiment are given by way of example only and are not intended to limit the present invention.

[0010] For example, in this disclosure, a case will be described where text data such as characters and sentences, which are information as a language model, is used, but the embodiment of the present invention is not limited to this and can also be applied to data such as images, audio, video, sensor data, etc. By appropriately tokenizing these data, they can be similarly applied in the forms described below.

[0011] Large-scale language models that use search results require a large amount of data related to the information to be searched. Furthermore, this data needs to cover a wide range of areas in the field to be searched, and it is important to consider how much of the data is used as a search result, i.e., how useful it is.

[0012] In order to obtain highly useful data, it is possible to give a contribution level to the provider of the data used in the search according to the usefulness of the data, and to provide incentives according to this contribution level. By setting an appropriate contribution level, it is possible to improve the overall quality of search data.

[0013] 1 is a diagram showing a schematic overview of a process according to an embodiment. The process in this diagram may be executed in an information processing system including, for example, one or more processors mounted on one or more computers and one or more storage devices (including the case where the storage devices are provided in the computers). The transmission and reception of each data and the storage location of each data can be set arbitrarily within the range where no inconsistency occurs depending on the system configuration.

[0014] In the following, the processing of at least one processor in the information processing system is described, but even if there is no particular limitation to at least one processor, it may indicate that at least one processor executes the processing, or it may indicate the processing flow of the information processing system as a whole. These may be appropriately interpreted within the scope of the technical ideas of this disclosure.

[0015] The information processing system is a system that generates and outputs third data, which is output data for first data, which is input data, using a first model, which is a probabilistic model trained using pre-learning data. The first model may be, for example, a base model or a generation model.

[0016] The first data is, for example, text for asking a question to the first model, and is called a prompt. Depending on the context, the first data can be interpreted as a question to which an answer is generated by the first model. In this disclosure, a "question" may conceptually include either a question from a user or an instruction from a user. Also, in this disclosure, a "question" may conceptually include either data input by a user or processed data obtained by performing a predetermined process on the data.

[0017] The first data may include any of images, audio, video, and sensor data as described above. The third data is, for example, text indicating an answer to a question. In the present disclosure, the "answer" may conceptually include any of an answer to a question and a response to an instruction. Also, in the present disclosure, the "answer" may conceptually include any of data generated by the first model and processed data obtained by performing a predetermined process on the data. The third data, like the first data, may include any of images, audio, video, and sensor data. Also, both the first data and the third data are not limited to these data, and may be data in other formats.

[0018] The first model is stored in a storage device in the information processing system and is formed by a processor in the information processing system by referring to the storage device. The information processing system executes inference using the first model formed by the processor.

[0019] The information processing system first receives first data as input data (S10). The input of data may be performed to at least one processor in the information processing system via various interfaces. For example, the at least one processor in the information processing system may receive the first data input by a user as the input data.

[0020] The first data input by the user is, for example, data input or specified by the user via a user interface of an information terminal, and may be text entered with a keyboard, text selected in a sentence, an image dragged and dropped with a mouse, or voice picked up by a microphone, etc. Here, the user's information terminal is an example of an information processing device and is one element constituting an information processing system.

[0021] At least one processor in the information processing system obtains second data for performing a search required to generate third data based on the first data received as input data (S12). The second data is, for example, a query to be input to a search engine. Information searched using the second data is used as additional input when determining the output of the third data. In other words, the second data is data for searching information related to an answer to the first data.

[0022] At least one processor in the information processing system may, for example, tokenize the first data by inputting the first data into a trained first model, and acquire a query required for a search as the second data based on the acquired tokens. As a non-limiting example, the first model may be a large-scale language model, and in this case, the processor acquires the second data by extracting a search query for generating third data from the tokenized first data for the first data indicating an input question sentence or the like.

[0023] Also, at least one processor in the information processing system may, for example, vectorize the first data and acquire the acquired vector as the second data which is a query required for the search. Also, at least one processor in the information processing system may, for example, acquire the first data itself as the second data which is a query required for the search. Also, at least one processor in the information processing system may use a conventional method to acquire the second data which is a query for a search engine described below based on the first data.

[0024] At least one processor in the information processing system searches for information required to generate the third data using a search engine, using the acquired second data as a query (S14). The search engine is stored in a storage device in the information processing system, similar to the first model, and is formed by at least one processor in the information processing system referencing this storage device. The information processing system executes a search for the required information using the search engine formed by the processor.

[0025] The information processing system inputs the second data into the search engine and obtains information to be used for generating the third data from the data group (S16). The processing of the search engine may be executed by at least one processor in the information processing system. By this processing, as a non-limiting example, the information processing system can obtain information to be used for generating an answer to the first data from the data group based on the first data in this processing.

[0026] The process by the search engine can be performed, for example, by using the data to be searched, which is information within a data group, as a key to extract a key for the query. Note that the search for the key by the search engine is not limited to a specific method, and any method for obtaining a key for a general query can be used.

[0027] The key and the query may be encoded. As an example, the search engine may execute a search using a function defined by the encoded key and the encoded query. The key and query encoding may also be pre-trained at the time of pre-training the first model.

[0028] As a non-limiting example, if the first model is a large-scale language model, at least one processor in the information processing system can generate a query required for the search from the tokenized text and use this query to reference keys in a data set to extract information that may include text required for an answer to the input text.

[0029] The search engine may be an engine formed by any method, such as a model formed by keyword matching, an attention mechanism, or a learning method based on reinforcement learning, etc. However, the search engine is not limited to these, and may be an engine formed by other methods.

[0030] At least one processor in the information processing system reflects the results of the search engine in generating output data in the first model (S18).

[0031] At least one processor in the information processing system generates third data based on the input data received in S10 and the information obtained by the search to generate the third data, which is output data in the first model (S20). That is, the at least one processor in the information processing system can use this information to generate an answer to the first data using the first model.

[0032] At least one processor in the information processing system may output the generated third data (answer) to the information terminal of the user who input the first data in S10. The output third data (answer) may be, for example, displayed on the user's information terminal, read aloud on the information terminal, printed via the information terminal, or the like.

[0033] At least one processor in the information processing system can, for example, input the input data and the information obtained by the search into the first model, and generate a response or answer using this information in addition to knowledge based on the pre-learned data as the third data. As an example, at least one processor in the information processing system can process the input data and the information obtained by the search into a prompt in a predetermined format, and input the prompt into the first model to generate the third data.

[0034] As another non-limiting example, at least one processor in the information processing system may input the information obtained by the search into the first model as an input to the internal calculation of the first model, and input the input data as a prompt into the first model to generate the third data. In this other example, the information input port is appropriately designed according to the architecture of the first model, and may be, for example, any of the input layer, intermediate layer, and output layer of the first model.

[0035] In this disclosure, inputting or incorporating information obtained by a search into a first model, or outputting third data from such a first model, may be referred to as augmenting the first model with search results (information obtained by a search) or augmenting the output of the first model with a search.

[0036] The third data is generated, for example, as data obtained by reinforcing the output of the first model for the first data through a search. The third data generated in this manner can be data that reflects newer information than, for example, data generated simply by using the first model.

[0037] The first model can generate the third data by using information of the encoded key set in the data group. As a non-limiting example, when the first model is a large-scale language model, the first model obtains information as a search result for the input text as an encoded embedding vector, and reflects the information of the encoded embedding vector in the output data, thereby generating a more accurate text, etc. as an answer to a question, etc.

[0038] At least one processor in the information processing system calculates and assigns a degree of contribution for information obtained from the data group and used to generate the third data (S22). The degree of contribution is an example of an index for evaluating the usefulness of information. The degree of contribution may be assigned at the time when the information is provided to the data group. Through this process, the information processing system can calculate, as a non-limiting example, the degree of contribution of the information in generating an answer by the first model.

[0039] The above is the overall process flow of the information processing system including the contribution degree calculation process according to the present disclosure. Next, the calculation of the contribution degree for each piece of information in a data group in the information processing system will be described in detail. The information processing system can calculate the contribution degree of the information at the timing when the information is provided to the data group to be searched or at the timing when the third data is generated.

[0040] 2 is a flowchart showing an example of a process of calculating the degree of contribution in the first stage. As the first stage of the contribution degree assignment, the information processing system can set the degree of contribution to the information at the timing when the first data is input and the third data is generated.

[0041] However, the information processing system may calculate the contribution degree of the first stage at a later timing than this timing. As yet another example, the information processing system may calculate the contribution degree of the first stage at the timing when it is determined that the data searched by the search engine in the first model is to be used for generating the third data.

[0042] The first stage of contribution assignment involves obtaining an evaluation that indicates the extent to which the information provided to the data set affects the distribution of third data generated by searching the data set.

[0043] In other words, when generating third data through augmentation with search, the contribution of this first stage can be used as a necessary indicator of the extent to which the information provided can have an impact on the overall data generated.

[0044] When a user attempts to obtain third data using the first model and the search engine, at least one processor in the information processing system can set a contribution to the provided information.

[0045] At least one processor in the information processing system accepts information provided by a data provider via a suitable interface (S100), such as, but not limited to, news articles, internal company documents including personnel and technical information, health information, image datasets, video datasets, experimental datasets, genomic data, and confidential information in general.

[0046] As with the input / output data, the information provided by the provider is not limited to text, but may be information that includes at least image, audio, video, sensor data, and other data.

[0047] The provider of this information may be, for example, a user related to the field of the data group, or in the case of a news article, the source of the news, or in the case of an internal company document, the company or department that provided the information.

[0048] According to a specification by a user who is a provider, the information processing system can accept the provision of data by automatically crawling information uploaded to a website existing in a specified database, storage device, or domain. In this case, in the case of information provided by other providers on a curation site, etc., the provider of the information referred to by the curation site can be regarded as the provider, and the curation site, etc. can be excluded from being regarded as the provider.

[0049] The provided information may be newly stored in a storage area within the information processing system, for example, or may be newly stored in a storage area outside the information processing system.Furthermore, the information processing system may refer to information in a storage area outside the information processing system (for example, a location specified by the user) where information is already stored.

[0050] At least one processor in the information processing system can store the information provided by the provider in any location. For example, the information may be stored in a storage area that requires security. In this case, the provider may grant the information processing system access to the information.

[0051] At least one processor in the information processing system may link the received information with various related information and manage it, such as the management data presented in an example, not limited to case (A) or case (B), in Figure 4.

[0052] At least one processor in the information processing system can, for example, as illustrated in (A), register in one record in a database, for each piece of provided information, information for uniquely identifying that information (information ID), information for uniquely identifying the provider of that information (provider ID), the number of times that information was used in generating an answer using the first model, and the total amount of royalties to be granted to the provider in accordance with the use of that information, thereby linking and managing the information ID, provider ID, number of uses, and total amount of royalties.

[0053] Furthermore, at least one processor in the information processing system can link and manage the date and time of use, the degree of contribution, the royalty amount, the question, and the answer by registering, in one record in a database, the date and time when the information was used in generating an answer by the first model (date and time of use), the degree of contribution of the information in generating an answer by the first model, the amount of royalty according to the degree of contribution, the question (first data) that triggered the use of the information, and the answer (third data) generated by the first model using the information, as illustrated in (B), for example.

[0054] At least one processor in the information processing system may acquire, for each piece of information, all or a part of the related information related to the piece of information by referring to the management data. At least one processor in the information processing system may provide all or a part of the related information related to the piece of information to a provider of the information or an information processing terminal of the provider, for example, in response to a request from the provider of the information or the provider's information processing device.

[0055] At least one processor in the information processing system accepts first data, which is input data from a user using the first model, and generates third data for the first data (S102). The at least one processor in the information processing system generates second data from the first data input by the user, and executes a search based on the second data using a search engine.

[0056] At least one processor in the information processing system calculates the contribution in the first stage based on a distribution of third data generated reflecting search results obtained by performing a search using the provided information as search target data, and a distribution of fourth data which is data generated without performing a search (S104). In other words, the at least one processor in the information processing system can calculate the contribution using the first model augmented with the search results and the first model not augmented with the search results.

[0057] At least one processor in the information processing system can calculate the contribution by, for example, comparing the first model augmented with the search results to the first model not augmented with the search results, where the fourth data does not need to be actually generated.

[0058] At least one processor in the information processing system calculates the contribution degree of the first stage using, for example, a distance between the distribution of the third data and the distribution of the fourth data, or a quantity equivalent to the distance. As a non-limiting example, the information processing system may calculate the contribution degree using the KL divergence (Kullback-Leibler Divergence) between the distribution of the third data and the distribution of the fourth data as a quantity equivalent to the distance. Of course, the information processing system can calculate the contribution degree using a distance or a statistical quantity other than these.

number

[0059] The KL divergence is expressed by formula (1), where P is the distribution of the third data generated by executing a search using a data group including the provided information, and P' is the distribution of the fourth data generated without executing a search. The information processing system can obtain the KL divergence by using sampled data based on a probability distribution.

[0060] At least one processor in the information processing system performs a search using the data set after information is provided for one prompt (first data) to obtain a sample S of third data according to a probability distribution P. At least one processor in the information processing system repeats this operation for the one prompt (first data) multiple times, for example, N times, and obtains samples S1, S2, ..., S according to the distribution P. N Get the.

[0061] At least one processor in the information processing system calculates a probability P(S i ) and the probability of each sample in the distribution of the fourth data P'(S i ) based on this, we can obtain the KL divergence as the first-stage contribution that indicates how much the provided data contributes.

number

[0062] At least one processor in the information processing system calculates the KL divergence by expressing the sampling result in a discretized expression such as equation (2). That is, the information processing system may calculate a value that approximates the KL divergence. Note that the value that approximates the KL divergence may be calculated using an equation other than equation (2). Sampling may be performed according to Monte Carlo sampling, for example.

[0063] FIG. 3 is a pseudocode showing an example of a process for obtaining the KL divergence based on this equation (2).

[0064] At least one processor in the information processing system initializes a variable sum indicating the KL divergence amount.

[0065] At least one processor in the information processing system extracts N samples from the distribution. The information processing system, for example, as described above, obtains the N samples by repeating the generation of third data (sample generation) for one prompt N times. As another non-limiting example, the information processing system may generate third data sufficiently more than N, rather than generating samples N times, and randomly extract N samples from the generated third data.

[0066] At least one processor in the information processing system calculates, for the extracted sample, a probability P(S i ), P'(S i ) is obtained for each of the extracted N samples. The difference d between the logarithms of the respective probabilities is calculated for the extracted N samples, and the sum sum of these differences d over the N samples is calculated to calculate the contribution of the first stage. The information processing system may also normalize the sum by dividing it by N.

[0067] Such KL divergence can be used as an index indicating that an output using data not included in the pre-training data used in training the first model can be provided via the first model as an output appropriately interpolated with newly provided information. In other words, the larger the value of this KL divergence, the more appropriately the information provider can provide data not included in the pre-training data.

[0068] Conversely, if the difference between the distribution (distribution shape) of the third data and the distribution (distribution shape) of the fourth data is small and the difference in the KL information is small, it indicates that the provided information is likely to have already been used in training. In other words, it means that even if a search is performed using the provided data, the same answer can be obtained as if the already trained first model were executed without performing a search process.

[0069] By setting the total number of samples N to a sufficiently large predetermined value, the KL divergence can be obtained as an approximation of formula (2). At least one information processing device in the information processing system may use this discretized KL divergence as the contribution degree of the first stage, or may calculate the contribution degree of the first stage by performing some kind of calculation, such as weighting, on the obtained KL divergence.

[0070] At least one processor in the information processing system may reflect the first-stage contribution to the provider who provided the information, and may store the provider (provider ID) and the information (information ID) in association with each other directly by one record or indirectly by multiple records (S106), as illustrated in, for example, Fig. 4. At least one processor in the information processing system may also increment the number of uses of the information acquired by the search in management data such as that shown in (A) of Fig. 4. The contribution to the provider may be accumulated, for example by adding up the contribution to the information provided up to that point, to be used as the total contribution to the provider in the first stage.

[0071] In addition, at least one processor in the information processing system can weight information so that older information is weighted lightly, or can delete information that has been around for a certain amount of time from the total contribution. By processing information along a time series in this way, it is possible to provide information that takes older information into consideration while weighting newer information.

[0072] As described above, when new information to be searched is provided to a data group, the information processing system according to the above embodiment can set an appropriate contribution degree for the provided data. Furthermore, the contribution degrees for each provider can be added up and processed. The information processing system can use the first-stage contribution degree as a necessary condition for the data provider as to whether the provided data can affect the output.

[0073] In the above, the probability distribution P of the third data is used as a starting point, but the present invention is not limited to this. For example, the probability distribution P' of the fourth data may be used as a starting point, and a sample P'(S i ') and this S i The third data for ' is sampled from the probability distribution P(S i The contribution may be calculated using the KL divergence KL(P'||P) obtained by

[0074] In addition, the provided information forming the probability distribution P with a high KL divergence may be used as learning data for re-learning the first model. By reflecting such data as learning data, it is possible to improve the accuracy of the output data in the first model.

[0075] Furthermore, the information processing system may repeatedly perform the process of FIG. 3 a number of times and calculate the degree of contribution based on the number of times the KL divergence amount becomes higher than a predetermined value.

[0076] The calculation of the contribution degree in the first stage is not limited to the above example. Several further examples are given below, but the calculation is not limited to these, and other methods that can reflect the complementarity of information when a search is appropriately performed may be used.

[0077] The information processing system can input a prompt (first data) to a first model augmented with search results to generate data (e.g., text), calculate the likelihood that this data will be output from the first model before augmentation (when search results are not used), and calculate the contribution of the first stage based on this likelihood. The information processing system can, for example, use the inverse of this likelihood as the contribution of the first stage.

[0078] The information processing system may generate data by inputting the prompt (first data) into each of a first model augmented with search results and a first model not augmented with search results, and input these two pieces of data into a model (e.g., a model trained by machine learning) that encodes data into latent variables, and calculate the first-stage contribution based on each latent variable. The information processing system may use an index such as the Euclidean distance between two latent variables in a latent space of latent variables, or a value obtained by subtracting the cosine similarity between the two latent variables from 1, as the first-stage contribution.

[0079] The information processing system can calculate the contribution of the first stage based on the cross entropy of the distribution P' of the fourth data to the distribution P of the third data when the prompt (first data) is input. For example, the information processing system may use this cross entropy as the contribution of the first stage, or may use a normalized amount of the cross entropy as the contribution of the first stage.

[0080] Next, a process of calculating the contribution degree of the second stage, which is the effect of the provided information on the answer as well as whether the data was useful for the answer, will be described. The information processing system may calculate the contribution degree of the second stage at the timing when the information is provided (at the timing of S104 in FIG. 2) in the same manner as the calculation of the contribution degree of the first stage described above, or, as another example, at the timing when the generation of the third data using the first model and a search engine is actually executed in response to a request from a user (at the timing of S22 in FIG. 1).

[0081] At least one processor in the information processing system can calculate the contribution of the second stage by evaluating whether the provided data actually contributed usefully to the result. This second stage evaluation can be realized, for example, by using a second model different from the first model. The information processing system can use, as an evaluator, for example, the second model, which is an evaluation model learned for evaluation, and which is trained as a model that returns a higher evaluation value when the input (first data) and output (third data or fourth data) of the first model are input, and the more correct the result is (the higher the correctness, validity, or validity).

[0082] For example, if the input of the first model is a question from a user, and the output of the first model is output as an answer to the question, the second model, which is an evaluator, may evaluate whether the output is an answer to the question, that is, the legitimacy, validity, or appropriateness of the output as an answer to the question. At least one of these evaluations may include, as an evaluation item, the correctness of the content of the output as an answer, or the degree to which the output maintains the format (style) of an answer, and in the case of natural language processing, the evaluator may evaluate each evaluation item by performing semantic analysis or contextual analysis.

[0083] Additionally, evaluating the validity, validity, or appropriateness of a generated answer conceptually includes evaluating the generated answer itself, as well as evaluating the answer by comparing the output from the first model that generated the answer, augmented with the search results (third data), with the output from the first model that is not augmented with the search results (fourth data).

[0084] At least one processor in the information processing system can, for example, use the second model to calculate the contribution as an expected value using a probability distribution P of third data generated using the provided data as search data and a probability distribution P' of fourth data generated without performing a search, as shown below.

number

[0085] Here, C(S) is the output value of the evaluator for sample S, and E(·) denotes the expected value. The information processing system performs the calculation using the expected value of C of sample S obtained according to probability distribution P, and the expected value of C of sample S' obtained according to probability distribution P'. More specifically, one or more samples are taken from the random variables S to P(S) and S' to P'(S'), and the expected values ​​of C(S) and C(S') are calculated using the average of the samples.

[0086] In this way, the information processing system can calculate the contribution of the second stage by sampling based on the probability distributions P and P' and calculating the expected value of the evaluation value. In other words, the information processing system can calculate the contribution using a first model reinforced with search results, a first model not reinforced with search results, and a second model that evaluates the answer generated by the first model. The second model that outputs the evaluation value can be trained by any method. As described above, the evaluation value calculated by this second model outputs a higher value as the answer is more correct.

[0087] Therefore, according to formula (3), the information processing system can calculate the difference between the expected value of the evaluation in the probability distribution of the third data generated by using the provided information for a search and the expected value of the evaluation in the probability distribution of the fourth data generated without performing a search as the contribution of the second stage.

[0088] In other words, the more correct the answer is when the provided information is used in a search, the greater the contribution of the second stage will be.

[0089] As described above, the information processing system of the above form can calculate a contribution reflecting the degree of legitimacy, validity, and appropriateness of the information provided in the data group in the answer, and how effective it is in obtaining a correct answer.

[0090] The information processing system can use both the first-stage contribution degree and the second-stage contribution degree, or can use only one of the contribution degrees. By setting the contribution degrees as described above, the information processing system can encourage information providers to provide information that improves the coverage rate of information in the first-stage contribution degree, and to encourage information providers to provide information that improves the legitimacy, validity, and validity of the data answered in the second-stage contribution degree.

[0091] The degree of contribution calculated above can be used, as a non-limiting example, as an index for realizing some processing, such as a payback as royalty to the information provider. The information processing system may, for example, grant to the information provider an amount of royalty (money or an equivalent remuneration) according to the degree of contribution of the information. Royalty is a concept that includes incentives, rewards, usage fees, points, etc. Granting royalties to the information provider may be done, for example, by linking the amount of royalty to the identification code of the information provider, or by other methods.

[0092] In the above, when the information processing system awards royalties to a provider of information that is obtained from a data set by a search and used to generate an answer using the first model, the information processing system determines the amount of the royalties according to the degree of contribution of the information. The method of awarding royalties is not limited to the above example.

[0093] For example, the information processing system may not calculate the contribution of information obtained from a data set by search and used to generate an answer using a first model, but may instead award a flat-rate royalty to the provider of the information regardless of the degree of contribution.

[0094] Alternatively, the information processing system may, for example, calculate the contribution of information obtained from a data set by a search and used to generate an answer using the first model, and if the contribution exceeds a certain level, grant a flat-rate royalty to the provider of the information.

[0095] That is, the information processing system may be configured to obtain a question (first data) to which an answer is generated by the first model, and based on that question, obtain by search information from a data set to be used to generate an answer to the question (third data), and grant royalties to the provider of the information.

[0096] The information processing system can be broadly divided into an answer generation system using the first model, a search system using a search engine, and a data set used by the search engine. The information processing system can provide any of these as an API (Application Programming Interface).

[0097] For example, as a whole, it is possible to provide a system that provides an API related to input and output of the first model, and obtains answers with search enhancement using information provided by an information provider as a search target.

[0098] For example, it is possible to provide a system that provides an API related to input and output of a search engine and uses information provided by an information provider as a search target.

[0099] For example, an API for accessing a data set can be provided, and a route for accessing data used in a search engine's search can be provided as a system.

[0100] In this way, the system can be divided arbitrarily. In any of the above-mentioned APIs, the information processing system according to the above-mentioned embodiment makes it possible to give an appropriate contribution level to the data provider.

[0101] That is, the information processing system may be in a form including at least one of the first model, the search engine, and at least a part of the sub-data group in the data group.

[0102] Furthermore, the information processing system may include, as multiple different information processing devices, an information processing device that generates the third data and an information processing device that calculates the contribution degree, or may be configured to realize the generation of the third data and the calculation of the contribution degree in the same information processing device. That is, the information processing system includes one or more information processing devices, and the processing in the present disclosure in these information processing devices can be appropriately distributed or aggregated and executed in the one or more information processing devices.

[0103] When the information processing system has a first model, the information processing device that sets the contribution degree can calculate the above-mentioned contribution degree from third data that is an output from the first model that uses data obtained by passing second data output from the first model held by the system to a search engine inside or outside the information processing system, and fourth data that is output from the first model without executing a search. An ID is assigned to each data (information) in a data group, and a contribution degree is set for this ID (information ID), and it is also possible to obtain data provided by the same provider from the ID and summarize their contribution degrees.

[0104] When the information processing system has a search engine, the information processing system that sets the degree of contribution can acquire the third data and the fourth data based on the output from the first model inside or outside the information processing system to set the degree of contribution. At the timing of generating the third data, the result of using the search engine in the information processing system can be passed to the first model.

[0105] When an information processing system has at least a part of a data group, the information processing system that sets the contribution level can obtain the third data and the fourth data using a first model and a search engine inside or outside the information processing system at the time the information is provided, set the contribution level, and assign it to each provided information of the data group.

[0106] The data set does not need to be stored together, and a part of the data may be stored in secret, and an API for accessing the encrypted key may be provided. For example, confidential data in multiple companies may be stored in secret in each company, and paths and interfaces for accessing the data may be prepared.

[0107] In this case, the sub-data groups that exist for each company can be concealed from the outside, so a user who wants an answer using the first model cannot detect which company's data is being used as the basis for the search. On the other hand, the information processing system can determine which sub-data group the answer is based on by referring to the ID (information ID) linked to the data in the data group.

[0108] In addition, the information processing system may output information indicating the provider of the information (information obtained by search) used to generate the answer using the first model (e.g., a provider ID, a string indicating the provider's name, and / or a string indicating the source or source of the information, such as a URL, etc.) in a manner that allows a user receiving the answer to identify that the answer was generated by reference to information provided by that provider.

[0109] Furthermore, the information processing system may output information indicating the provider of the information (information acquired by search) used to generate the answer by the first model (e.g., a provider ID, a character string indicating the provider's name, and / or a character string indicating the source or acquisition location of the information, such as a URL, etc.) in a manner (e.g., outputting the information together with the answer) that enables a user who receives the answer (a recipient of the answer) to identify that the answer was generated by referring to (using) the information provided by the provider. The information indicating the provider may be text, an image, etc.

[0110] The information processing system can also assign a level of contribution to information providers who have groups of sub-data sources identified in this way during actual operation, based on the number of times and frequency that the data was used as an answer.

[0111] In either case, the first model may have various forms. As described above, the large-scale language model may be a model that inputs text such as sentences or words and outputs text such as sentences or words. As another example, the first model may be a model that outputs an image or video when text is input, a model that outputs text when an image or video is input, a model that inputs an image or video when an image or video is input, a model that outputs graph information when graph information is input, a model that outputs text or an image or video when graph information is input, a model that outputs graph information when text or an image or video is input, or a model that outputs graph information when text or an image or video is input. Of course, the first model may be a model other than these.

[0112] The trained model in the above embodiment may be a concept that includes, for example, a model that has been trained as described and then further distilled using a general method.

[0113] In the following, each device in the embodiment may be at least one device present in the information processing system. The device may be, for example, a device that sets a contribution degree at the timing when information is provided to a data group in the information processing system, or a device that sets a contribution degree at the timing when third data is generated, or a device that uses a first model, or a device that uses a search engine, or a device that performs at least a plurality of operations among them.

[0114] A part or all of each device (information processing device provided in the information processing system) in the above-mentioned embodiment may be configured with hardware, or may be configured with information processing of software (program) executed by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) or the like. In the case of being configured with information processing of software, software that realizes at least a part of the functions of each device in the above-mentioned embodiment may be stored in a non-transient storage medium (non-transient computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the information processing of the software may be executed by reading the software into a computer. The software may also be downloaded via a communication network. Furthermore, the information processing by the software may be executed by hardware by implementing all or a part of the processing of the software in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0115] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk or memory. The storage medium may be provided inside the computer (such as a main storage device or auxiliary storage device) or outside the computer.

[0116] 5 is a block diagram showing an example of the hardware configuration of each device (information processing device provided in the information processing system) in the above-mentioned embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76.

[0117] Although the computer 7 in FIG. 5 includes one of each component, it may include multiple of the same components. Although one computer 7 is shown in FIG. 5, the software may be installed in multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, it may be a form of distributed computing in which each computer communicates via a network interface 74 or the like to execute the processing. In other words, each device (information processing device provided in the information processing system) in the above-mentioned embodiment may be configured as a system that realizes a function by one or more computers executing instructions stored in one or more storage devices. Also, it may be configured such that information transmitted from a terminal is processed by one or more computers provided on the cloud, and the processing results are transmitted to the terminal.

[0118] Various calculations of each device (information processing device provided in the information processing system) in the above-mentioned embodiment may be executed in parallel using one or more processors, or using multiple computers via a network. Also, various calculations may be distributed to multiple arithmetic cores in a processor and executed in parallel. Also, a part or all of the processes, means, etc. disclosed herein may be realized by at least one of a processor and a storage device provided on a cloud that can communicate with the computer 7 via a network. Thus, each device in the above-mentioned embodiment may be in the form of parallel computing using one or more computers.

[0119] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls a computer or performs calculations. The processor 71 may be any of a general-purpose processor, a dedicated processing circuit designed to execute a specific calculation, or a semiconductor device including both a general-purpose processor and a dedicated processing circuit. The processor 71 may include an optical circuit, or may include a calculation function based on quantum computing.

[0120] The processor 71 may perform arithmetic processing based on data or software input from each device or the like configured inside the computer 7, and may output arithmetic results or control signals to each device or the like. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, or the like.

[0121] Each device (information processing device provided in the information processing system) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other by wire or wirelessly.

[0122] The main memory device 72 may store instructions executed by the processor 71 and various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memories may be either volatile memories or non-volatile memories. The memory devices for saving various data, etc. in each device (information processing device provided in the information processing system) in the above-mentioned embodiment may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by an internal memory built into the processor 71. For example, the memory unit of the information processing device provided in the information processing system in the above-mentioned embodiment may be realized by the main memory device 72 or the auxiliary memory device 73.

[0123] When each device (information processing device provided in the information processing system) in the above-mentioned embodiment is composed of at least one storage device (memory) and at least one processor connected (coupled) to the at least one storage device, at least one processor may be connected to one storage device. At least one storage device may be connected to one processor. A configuration in which at least one processor among a plurality of processors is connected to at least one storage device among a plurality of storage devices may also be included. This configuration may also be realized by storage devices and processors included in a plurality of computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may also be included.

[0124] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. The network interface 74 may be an appropriate interface, such as one conforming to an existing communication standard. The network interface 74 may exchange information with an external device 9A connected via the communication network 8. The communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0125] The device interface 75 is an interface such as a USB that directly connects to an external device 9B.

[0126] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device directly connected to the computer 7.

[0127] The external device 9A or the external device 9B may be, for example, an input device. The input device is, for example, a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device having an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0128] Moreover, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the output device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0129] In addition, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage device such as a HDD.

[0130] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (information processing device provided in the information processing system) in the above-mentioned embodiment. In other words, the computer 7 may transmit some or all of the processing results to the external device 9A or the external device 9B, or may receive some or all of the processing results from the external device 9A or the external device 9B.

[0131] In this specification (including the claims), when the phrase "at least one of a, b, and c" or "at least one of a, b, or c" (including similar phrases) is used, it includes any of a, b, c, a-b, a-c, b-c, or a-b-c. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, etc. It also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in a-b-c-d.

[0132] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes the use of data itself or data that has been processed in some way (e.g., data with noise added, normalized, features extracted from data, intermediate representation of data, etc.). In addition, when a statement is made that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), unless otherwise specified, this includes the use of the result based only on the data in question or the use of the result influenced by other data, factors, conditions, and / or states other than the data in question. In addition, when a statement is made that "data is output" (including similar expressions), unless otherwise specified, this includes the use of data itself as output or data that has been processed in some way (e.g., data with noise added, normalized, features extracted from data, intermediate representation of data, etc.) as output.

[0133] In this specification (including the claims), the terms "connected" and "coupled" are intended as open-ended terms including any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, physically connection / coupling, etc. The terms should be interpreted appropriately according to the context in which the terms are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in the terms without any limitation.

[0134] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured / set to actually perform operation B. For example, when element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, when element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0135] When used in this specification (including the claims), terms implying containing or possessing (e.g., "comprising / including" and "having") are intended to be open-ended terms that include the inclusion or possession of things other than the object indicated by the object of the term. When the object of such terms implies no quantity or a singular number (such as an article such as "a" or "an"), the expression should be construed as not being limited to a specific number.

[0136] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles a or an) should be interpreted as not necessarily being limited to a specific number.

[0137] In this specification, when a particular advantage / result is described as being obtained from a particular configuration of an embodiment, it should be understood that the same effect can also be obtained from one or more other embodiments having the same configuration, unless there is a reason to the contrary. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect is not necessarily obtained in the invention according to the claim that specifies the configuration or a similar configuration.

[0138] In this specification (including the claims), when a term such as "maximize" is used, it includes finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted accordingly according to the context in which the term is used. It also includes finding approximations of these maxima probabilistically or heuristically. Similarly, when a term such as "minimize" is used, it includes finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted accordingly according to the context in which the term is used. It also includes finding approximations of these minima probabilistically or heuristically. Similarly, when a term such as "optimize" is used, it includes finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted accordingly according to the context in which the term is used. It also includes finding approximations of these optimum values ​​probabilistically or heuristically.

[0139] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or a part of the hardware may perform all of the predetermined process. Also, a part of the hardware may perform a part of the predetermined process, and another piece of hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. The hardware may include an electronic circuit, or a device including an electronic circuit.

[0140] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0141] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, replacements, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents. For example, in the above-mentioned embodiments, when numerical values ​​or formulas are used for explanation, these are shown for illustrative purposes and do not limit the scope of the present disclosure. In addition, the order of each operation shown in the embodiment is also illustrative and does not limit the scope of the present disclosure.

Claims

1. one or more memories; one or more processors; Equipped with The one or more processors: Obtain a question for which an answer is generated by the first model; obtaining information from a set of data based on the question for use by the first model in generating an answer to the question; calculating a contribution of the information to generating the answer by the first model; Information processing device.

2. The one or more processors grant royalties to the provider of the information in an amount corresponding to the degree of contribution of the information. The information processing device according to claim 1.

3. one or more memories; one or more processors; Equipped with The one or more processors: Obtain a question for which an answer is generated by the first model; obtaining information from a set of data based on the question for use by the first model in generating an answer to the question; granting royalties to the provider of said information; Information processing device.

4. the one or more processors award the royalty to the information provider in accordance with a contribution of the information to generation of the answer by the first model; The information processing device according to claim 3.

5. The one or more processors: outputting information indicating a provider of the information used in generating the answer using the first model in a manner that allows a recipient of the answer to identify that the answer was generated using information provided by the provider; 5. An information processing device according to claim 1.

6. the one or more processors calculate the contribution using the first model augmented with the information and the first model not augmented with the information; 5. An information processing device according to claim 1, 2 or 4.

7. The first model is a probabilistic model.

5. An information processing device according to claim 1, 2 or 4.

8. the first model is a language model; 5. An information processing device according to claim 1, 2 or 4.

9. The one or more processors: using the information to generate an answer to the question using the first model; 5. An information processing device according to claim 1, 2 or 4.

10. The one or more processors: obtaining second data for retrieving information used by the first model in generating an answer to the question; inputting the second data into a search engine and executing a search to obtain the information from the data set; 5. An information processing device according to claim 1, 2 or 4.

11. the second data is data obtained by inputting the question into the first model; The answer is data generated by inputting information acquired using the search engine based on the second data into the first model. The information processing device according to claim 10.

12. The one or more memories include not storing the first model; The information processing device according to claim 1.

13. The one or more memories include storing the first model; The information processing device according to claim 1.

14. The one or more processors include at least calculating the degree of contribution to be higher when data generated using the first model augmented with the information is more legitimate than data generated using the first model not augmented with the information; The information processing device according to claim 2.

15. The one or more processors include at least Calculating the contribution to the provided information based on a distribution of data generated using the first model augmented with the information and a distribution of data generated using the first model not augmented with the information. The information processing device according to claim 2.

16. The one or more processors: Calculating the contribution of the information for each provider of the provided information; An information processing device according to any one of claims 1 to 4 or any one of claims 12 to 15.

17. At least a part of the data in the data group is managed in confidentiality by the provider of the data. The information processing device according to claim 16.

18. One or more information processing devices are provided, The one or more information processing devices include Obtain a question for which an answer is generated by the first model; obtaining information from a set of data based on the question for use by the first model in generating an answer to the question; Calculating a contribution of the information to generating the answer by the first model; The one or more information processing devices that generate an answer to the question and the one or more information processing devices that calculate the contribution of the information are different information processing devices, or at least a part of them are the same information processing device. Information processing system.

Images