Information processing device and information processing system

Abstract

To set an appropriate degree of contribution according to provided data.SOLUTION: An information processing device includes one or more memories and one or more processors. The one or more processors each acquire a question for which an answer is generated by a first model, acquire information, which is used by the first model in generation of an answer to the question, from a data group on the basis of the question, and calculate a degree of contribution of the information in generating the answer by the first model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing system. [Background technology]

[0002] Large-scale language modeling (LLM), a language model trained by machine learning, is Large-scale language models (LMS) are beginning to be used for a variety of purposes. One use case is to search for information on inputs and generate outputs using large language models. A technique called Retrieval Augmented Generation (RAG) that uses the retrieved information It is known that. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] S. Borgeaud, et.al., “Improving language models by retrieving from trillions of tokens”, arXiv.org:2112.04426v3, 7 February 2022, https: / / arxiv.org / pdf / 2112.04426v3.pdf Summary of the Invention [Problem to be solved by the invention]

[0004] One non-limiting problem that the embodiments of the present disclosure aim to solve is the problem of large-scale language models, etc. The aim of this study is to evaluate the information used to generate output from the underlying and generative models. [Means for solving the problem]

[0005] According to one embodiment, an information processing device includes one or more memories and one or more processors. and wherein the one or more processors generate a question for which an answer is generated by the first model. and generating an answer to the question from a set of data based on the question. and obtaining information used by a model in generating the answer by the first model. The contribution of the information is calculated.

[0006] According to one embodiment, an information processing system includes one or more information processing devices. The plurality of information processing devices generate first data corresponding to a question for which an answer is generated by the first model. and, based on the question first data, select a response to the question first data from a data group. and obtaining information used by the first model to generate an answer to the problem. A contribution of the information to generating an answer is calculated. and one or more of the information processing devices that calculate the contribution of the information. The information processing device is a different information processing device or is at least partially the same information processing device. It is. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an outline of a process according to an embodiment. [Diagram 2] FIG. 11 is a diagram showing an example of a contribution degree calculation process according to an embodiment. [Diagram 3] FIG. 13 is a diagram showing a pseudo code of an example of a contribution calculation process according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of information management according to an embodiment. [Diagram 5] FIG. 1 is a diagram showing an implementation example of an information processing device constituting at least a part of an information processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The problems to be solved by the embodiments of the present disclosure are not limited to the problems described above. As examples of some problems described in the embodiments, but not limited thereto, It can also be said that the problem corresponds to the effect. The present disclosure aims to solve the problem corresponding to at least one of the effects described above. The objective of the present invention is to

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. It is provided by way of example and not by way of limitation.

[0010] For example, in the present disclosure, text such as characters and sentences, which are mainly information as a language model, Although the case where the data of the data is used will be described, the embodiment of the present invention is not limited to this. The present invention can be applied not only to image data, but also to image, audio, video, sensor data, and other data. These data are appropriately tokenized and can be viewed in the form described below. can be applied in the same way.

[0011] Large-scale language models that utilize search results require a large amount of data related to the information being searched. Furthermore, this data needs to cover a wide area in the field being searched. It is necessary to know how much of the data was used as a search result, i.e., how useful the data is. What is important is whether the data was correct.

[0012] In order to obtain highly useful data, we ask the providers of the data used in the search to provide us with the usefulness of the data. It is also possible to give a contribution level according to the contribution of each employee and provide incentives according to this contribution level. By setting appropriate contribution levels, you can improve the overall quality of your search data. It becomes.

[0013] FIG. 1 is a diagram showing an outline of a process according to an embodiment of the present invention. For example, one or more processors installed in one or more computers, and one or more In an information processing system having a number of storage devices (including those provided in a computer) The transmission and reception of each data and the storage location of each data may be determined by the system. The value can be set arbitrarily as long as no contradiction occurs depending on the system configuration.

[0014] In the following, the processing will be described as processing of at least one processor in an information processing system. However, even if there is no particular limitation to at least one processor, It can also refer to a single processor performing the processing, or it can refer to an information processing system. These may represent the flow of processing for the entire system. It can be interpreted appropriately within the scope of the concept.

[0015] The information processing system uses the first model, which is a probabilistic model trained using pre-training data. Generate third data, which is output data for the first data, which is input data, using The first model may be, for example, a base model or a generative model. stomach.

[0016] The first data is, for example, a text for asking a question to the first model, and a prompt The first data is the question for which the first model generates an answer, depending on the context. In this disclosure, the term "question" refers to a question from a user and a response from a user. In the present disclosure, a "question" may conceptually include any of the following instructions from a user: The input data and the processed data obtained by performing a specified process on the input data. It may conceptually include deviations.

[0017] The first data may include any of image, audio, video, and sensor data as described above. The third data is, for example, a text showing an answer to a question. The term "answer" may conceptually include both an answer to a question and a response to an instruction. In this disclosure, an "answer" refers to the data generated by the first model and the The third data conceptually includes both processed data and data obtained by performing a certain process. The data may include any of images, audio, video, and sensor data, similar to the first data. In addition, the first data and the third data are not limited to these data and may be in other forms. It may be formula data.

[0018] The first model is stored in a storage device in the information processing system, and The information processing system is formed by the processor referring to this storage device. Inference is then performed using the first model formed by the above.

[0019] The information processing system first receives the first data as input data (S10). The input is input to at least one processor in the information processing system via various interfaces. At least one processor in the information processing system may execute For example, first data input by a user may be received as the input data.

[0020] The first data input by the user is, for example, a user input through a user interface of an information terminal. Data entered or specified via the Internet, such as text entered on a keyboard, Selected text in the Here, the user's information terminal is an example of an information processing device. , is one element that constitutes an information processing system.

[0021] At least one processor in the information processing system processes the first data received as input data. Based on the second data, to carry out the search necessary to generate the third data. The second data is, for example, a query to be input to a search engine. The information retrieved using this second data is used as additional input in determining the output of the third data. That is, the second data is used to search for information related to a response to the first data. This is data for.

[0022] At least one processor in the information processing system, for example, converts the first data into a learned Tokenize the first data by inputting it into a model, and perform a search based on the obtained tokens As a non-limiting example, the first model may acquire a query required for the first model. The model may be a large language model, in which case the processor may generate a large number of For the first data indicating the above, a search query for generating the third data is tokenized. The second data is obtained by extracting it from the first data.

[0023] Also, at least one processor in the information processing system may, for example, store the first data in a vector. The vector is then converted into a second data set, which is the query required for the search. Also, at least one processor in the information processing system may, for example, The information itself may be acquired as second data, which is a query required for a search. At least one processor in the processing system performs a query to a search engine as described below. Conventional methods may be used to obtain the second data, where .times. ...

[0024] At least one processor in the information processing system queries the obtained second data. Then, the search engine is used to search for information required to generate the third data (S14 The search engine is stored in the storage device of the information processing system, just like the first model, and At least one processor in the information processing system references the storage device. The information processing system uses a search engine formed by a processor to Perform a search for relevant information.

[0025] The information processing system inputs the second data into the search engine, and extracts the third data from the data group. The search engine process is performed by the information processing system. This process may be performed by at least one processor in the system. In this process, the processing system may, as a non-limiting example, generate a data Information used to generate a response to the first data can be obtained from the data group.

[0026] The process by the search engine is carried out by, for example, searching target data, which is information within a data set, as a key. This can be done by extracting the key for the query. The search for keys by is not limited to a specific method, and it is possible to find keys for general queries. The method of obtaining the information can be used.

[0027] Additionally, the key and the query may each be encoded. The method performs a search using a function defined by an encoded key and an encoded query. The key and query encoding can also be done at the time of pre-training the first model. You can also study these topics in advance.

[0028] As a non-limiting example, when the first model is a large-scale language model, the information processing system At least one processor in the system extracts the necessary queries from the tokenized text. By using this query to look up the keys in the data set, the input text is The system can extract information, which may include text, that is needed to respond to a question.

[0029] This search engine also uses, for example, keyword matching and attention mechanisms. A model formed by any method, such as a model formed by a learning method based on reinforcement learning, The engine may be an engine formed by a law, but is not limited to these. The present invention is not limited to engines formed by other techniques.

[0030] At least one processor in the information processing system processes the results of the search engine. This is reflected in the generation of output data in the first model (S18).

[0031] At least one processor in the information processing system processes the input data received in S10, and and the third data, which is the output data in the first model, is obtained by searching. Based on the information, third data is generated (S20). At least one processor uses this information to generate an answer to the first data using the first model. can be generated.

[0032] At least one processor in the information processing system generates third data (answer). The third data may be output to the information terminal of the user who input the first data in S10. The answer can be displayed on the user's information terminal or read aloud on the information terminal. The information may be uploaded to a terminal, printed out via an information terminal, etc.

[0033] At least one processor in the example information processing system may, for example, receive input data and The information obtained by the search is input to the first model, which generates knowledge based on the pre-trained data. In addition to the identification, a response or answer can be generated using this information as third data. For example, at least one processor in the information processing system may receive input data and The information obtained by the search is processed into a prompt in a specified format, and this prompt is The data may be input into the first model to generate third data.

[0034] As another non-limiting example, at least one processor in the information processing system may The information obtained by the search is fed into the first model as input to the first model's internal calculations; The first model may be input with the input data as a prompt to generate the third data. In another example, the information inlet is appropriately designed according to the architecture of the first model. For example, it may be any one of the input layer, intermediate layer, and output layer of the first model.

[0035] In the present disclosure, the information obtained by the search is input to or added to a first model. and importing or outputting third data from such a first model. Augment the first model with the results (information obtained by searching) or search the output of the first model This is sometimes referred to as "reinforcing by."

[0036] The third data is, for example, data obtained by augmenting the output of the first model for the first data with a search. The third data thus generated is, for example, simply the first model. This allows the data to reflect more up-to-date information compared to when the data is used.

[0037] The first model uses the encoded key information set in the data set. Third data may be generated. As a non-limiting example, the first model may be a large-scale language model. If the input text is a string, the first model returns information as a search result for the input text. The information of the encoded embedding vector is obtained as By reflecting this information in the output data, more accurate text can be generated as answers to questions, etc. It is possible to generate, for example,

[0038] At least one processor in the information processing system generates a third data set obtained from the data set. The contribution level of the information used to generate the data is calculated and assigned (S22). The degree of contribution is an example of an index for evaluating the usefulness of information. By this process, the information processing system As a non-limiting example, calculating the contribution of information in generating an answer by the first model It is possible.

[0039] The above is the overall process flow of the information processing system including the contribution degree calculation process according to the present disclosure. Next, the contribution of each piece of information in the data set in the information processing system is calculated. The calculation will be explained in detail. The information processing system calculates information for the data set to be searched. At the time when the information is provided or the third data is generated, The contribution of the information can be calculated.

[0040] FIG. 2 is a flowchart illustrating an example of a first-stage process of calculating the contribution degree. The system is a type in which the first data is input and the third data is generated as the first stage of contribution awarding. You can set the contribution level for that information by tapping.

[0041] However, the information processing system will contribute to the first stage at a later time. In yet another example, the information processing system may calculate a degree of The Thai government decided to use data searched by a search engine in generating third-party data. It is also possible to calculate the first stage contribution using

[0042] The first stage of contribution is the assignment of contribution points to the data set, which is generated by searching the data set based on the information provided to the data set. The purpose of this study is to obtain an evaluation showing the extent to which the third data distribution is affected by the change in the do.

[0043] In other words, when generating the third data by augmenting it with search, the contribution of this first stage is the extent to which the information provided can affect the overall data generated. This can be an indication of the requirements to be met.

[0044] The timing at which the user attempts to obtain the third data using the first model and a search engine At least one processor in the information processing system performs the following on the provided information: The contribution level can be set by the

[0045] At least one processor in the information processing system is connected to the Information provided by a data provider is received (S100). The information may be, for example, news articles. , Internal documents including personnel and technical information, health information, image datasets, video datasets This may be information such as experimental data sets, genomic data, or confidential information in general. However, the present invention is not limited to these.

[0046] As with input and output data, the information provided by the provider is not limited to text. The information may include at least data such as images, audio, video, and sensor data.

[0047] The providers of this information may be, for example, users in the field of the data set; If it is a news article, the news source, etc. If it is an internal document, the company, etc. that provided it The department in charge may be the one in charge of the management.

[0048] The information processing system uses a specified database according to the specification of the provider, the user. Automatically transfer information uploaded to a website located on a storage device or domain It is also possible to accept data by crawling the In the case of information provided by other providers on websites, etc., In this case, the source of the information referenced by the curation site is the provider, and the curation site is the It is also possible to exclude application sites, etc. from being providers.

[0049] The provided information may be newly stored in a storage area in an information processing system, for example. Alternatively, the information may be newly stored in a storage area outside the information processing system. The information processing system may access a storage area outside the information processing system where information is already stored (e.g. For example, the information may be in a form that refers to a location specified by the user.

[0050] At least one processor in the information processing system thus processes the information provided by the provider. The information can be stored anywhere. For example, the information can be stored in a secure In this case, the provider may provide access to the information to the information processing system. This can also be implemented by allowing access to

[0051] At least one processor in the information processing system is in the case of (A) or (B) in FIG. The information received and various related information, such as the management data presented in the following non-limiting examples, The relevant information may be linked and managed.

[0052] At least one processor in the information processing system may, for example, be as illustrated in (A). For each piece of information provided, information to uniquely identify that piece of information (information ID), Information to uniquely identify the information provider (provider ID), used to generate answers using the first model The number of times the information is used in the By registering the total amount in one record in the database, the information ID, provider ID, number of uses, The total royalty amount can be linked and managed.

[0053] In addition, at least one processor in the information processing system may be, for example, The date and time when the information was used in generating the answer by the first model (the usage date and time), The contribution of the information to the generation of the answer by the first model, and the royalty based on that contribution the amount of the money, the question that prompted the information to be used (first data), and the first The answer generated by the model (third data) is registered as one record in the database. This allows you to link and manage usage dates and times, contributions, royalty amounts, questions, and answers. do.

[0054] At least one processor in the information processing system performs, for each piece of information, All or part of the related information may be acquired by referring to the management data. At least one processor in the system may be, for example, a provider of information or a provider's information processing device. Upon request from the relevant authority, we may provide all or part of such related information to the relevant authority. It may be provided to the donor or the provider's information processing terminal.

[0055] At least one processor in the information processing system receives a signal from a user using the first model. The first data is received as input data, and third data is generated for the first data (S 102) At least one processor in the information processing system processes the first data input by the user. and generating second data from the data and performing a search based on the second data using the search engine.

[0056] At least one processor in the information processing system processes the provided information into a search target database. The distribution of the third data generated by reflecting the search results obtained by using the data as the search data and the search results Based on the distribution of the fourth data, which is data generated without execution, the contribution in the first stage is calculated. In other words, at least one process in the information processing system calculates the contribution (S104). The first model is augmented with search results, the second model is not augmented with search results, and The contribution can be calculated using:

[0057] At least one processor in the information processing system may, for example, generate a search result augmented with The contribution is calculated by comparing the first model with the first model that is not augmented with search results. Here, the fourth data does not have to be actually generated.

[0058] At least one processor in the information processing system, for example, 4 The contribution of the first stage is calculated using the distance from the distribution of the data or a quantity equivalent to the distance. As a non-limiting example, the information processing system may include a distribution of the third data and a distribution of the fourth data. Contribution is calculated using the KL information (Kullback-Leibler Divergence) as a measure of distance. Of course, the information processing system may use other distances or statistics to calculate the contribution. The contribution can be calculated.

number

[0059] The KL divergence is expressed by the formula (1). Here, P is the number of predictions using the data set including the provided information. The distribution of the third data generated by performing a search, P', is the distribution of the fourth data generated without performing a search. The data distribution is shown. The information processing system samples the data based on the probability distribution. By using this, the KL divergence can be obtained.

[0060] At least one processor in the information processing system receives a prompt (first data). Then, a search is performed using the data set after the information is provided, and the probability distribution P is used to and acquiring a third sample S of data. repeats this process for one prompt (first data) multiple times, for example, N times. Let S1, S2, ..., S follow the distribution P. N Get the.

[0061] At least one processor in the information processing system The probability of the sample being i ) and the probability of each sample in the distribution of the fourth data P'(S i ) and the first-stage contribution level, which indicates how much the provided data contributes based on The KL divergence can be obtained as

number

[0062] At least one processor in the information processing system calculates the sampled result by the following equation (2): By expressing it in a discretized form like this, the KL divergence is calculated. The KL divergence may be approximated by using formula (2) and another formula. The sampling may be performed, for example, according to Monte Carlo sampling. It may be performed as follows.

[0063] FIG. 3 is a pseudocode showing an example of a process for obtaining the KL divergence based on this equation (2).

[0064] At least one processor in the information processing system initializes a variable sum indicating the KL divergence. do.

[0065] At least one processor in the information processing system is configured to draw N samples from the distribution. The information processing system may, for example, generate third data for one prompt as described above. By repeating the generation (sample generation) of N times, we obtain N samples. As another example, instead of generating N samples, the information processing system may generate N samples. Generate a sufficiently large number of third data, and randomly select N samples from the generated third data. A sample may be extracted.

[0066] At least one processor in the information processing system performs, on the extracted sample: The probability P(S i ) , P'(S i ) for each of the N samples. The contribution of the first stage is calculated by calculating the difference d and then calculating the sum of this difference d over N samples. The information processing system may also normalize the sum by dividing it by N. .

[0067] Such KL divergence can be calculated, for example, by using the pre-training data used in training the first model. The output using data not included in the table is appropriately interpolated with the newly provided information. This can be used as an indicator that the first model can provide the The larger the KL divergence, the more information is provided that is not included in the pre-training data. It can be said that the person in charge is providing the information appropriately.

[0068] Conversely, the difference between the distribution (distribution shape) of the third data and the distribution (distribution shape) of the fourth data is small. If the difference in the KL divergence is small, the provided information has already been used in training. This indicates that there is a high probability that a search using the provided data will yield results that have already been found. The first model, which is trained on the It means you can get it.

[0069] By setting the total number of samples N to a sufficiently large value, the KL divergence can be calculated as an approximation of equation (2). At least one information processing device in the information processing system performs this discretization. The KL divergence obtained may be used as the contribution of the first stage, or the obtained KL divergence may not have any effect. By performing some calculation, such as weighting, the contribution of the first stage is calculated. Good too.

[0070] At least one processor in the information processing system may, for example, be as illustrated in FIG. The contribution of the first stage is reflected to the provider who provided the information, and this provider (provider ID) and ,The information (information ID) and ,are directly linked to one record or to multiple records. In addition, the information processing system may indirectly associate and store the information by using the In each case, a processor retrieves the management data as shown in FIG. The contribution to the provider of the information provided may be increased by one. The contribution of the information provided up to that point is added together to the contribution of the information provided up to that point, and the contribution is then cumulatively calculated as the first It can also be expressed as the total contribution of the stage.

[0071] At least one processor in the information processing system weights older information more heavily. Weighting may be done, such as making the information lighter, or information that has been around for a certain amount of time may be included in the total contribution. By processing in chronological order like this, old It is also possible to provide information that takes into consideration the latest information, but gives more weight to it.

[0072] As described above, the information processing system according to the above embodiment can provide a new search target for a data group. When relevant information is provided, an appropriate contribution level is set for the provided data. Furthermore, the contributions of each provider can be added together for processing. The processing system informs the data provider whether the provided data can affect the output. The contribution from the first stage can be used as a necessary condition for whether or not a

[0073] In the above, the probability distribution P of the third data is used as a starting point, but this is not limited to this. For example, starting from the probability distribution P' of the fourth data, we can generate a sample P'(S i ') and this S i The third data for ' is sampled from the probability distribution P(S i ') The contribution may be calculated using the obtained KL divergence KL(P'||P).

[0074] In addition, the provided information that forms the probability distribution P with a high KL divergence is used as the learning information for retraining the first model. By reflecting such data as training data, It is also possible to improve the accuracy of the output data in the first model.

[0075] The information processing system repeats the process of FIG. 3 multiple times to determine whether the KL divergence is equal to a predetermined value. The degree of contribution may be calculated based on the number of times that the value is higher than the value.

[0076] The calculation of the contribution in the first stage is not limited to the above example. Here are some examples that may be useful, but are not limited to these. Other methods that can reflect the complementarity of the information in question may also be used.

[0077] The information processing system converts the prompt (first data) into a first model augmented with search results. Input generates data (e.g., text), and this data is used for pre-enhancement (without using search results). Calculate the likelihood of the output from the first model (if the first model is not used), and calculate the contribution of the first stage based on this likelihood. The information processing system can calculate, for example, the inverse of this likelihood as the contribution of the first stage. It can be considered a dedication.

[0078] The information processing system processes the prompt (first data) into the first model and the second model augmented by the search results. The data was generated by inputting the first model, which was not augmented with the search results, into these two models. A model that encodes data into latent variables (as an example of machine learning training) We also calculated the first-stage contributions based on each latent variable. The information processing system may, for example, determine the unity between two latent variables in the latent space of latent variables. The arithmetic distance, or the cosine similarity between two latent variables minus 1, The indicator can be the first level of contribution.

[0079] The information processing system classifies the third data when the prompt (first data) is input. Calculate the contribution of the first stage based on the cross entropy of the distribution P' of the fourth data for fabric P. The information processing system can use this cross entropy as the first step. The contribution can be calculated as the cross-entropy standardized value, or the contribution of the first stage can be calculated as the cross-entropy standardized value. good.

[0080] Second, the information provided was useful data for the answer, in addition to its impact on the answer. The process of calculating the degree of contribution in the second stage based on whether the information processing system is The second stage of the contribution calculation is carried out in the same way as the first stage of the contribution calculation above, for the categories for which information was provided. As another example, the user may actually The timing of generating the third data using the first model and the search engine upon request from This may be performed during the processing (timing S22 in FIG. 1).

[0081] At least one processor in the information processing system processes the provided data to produce an actual result. The contribution of the second stage can be calculated by evaluating whether the second stage contributed usefully to the The evaluation of the floor can also be achieved, for example, by using a second model different from the first model. The information processing system is, for example, an evaluation model trained for evaluation, and the input of the first model ( When the first data) and the output (third or fourth data) are input, the result is correct (validity, The higher the effectiveness or validity of the model, the higher the evaluation value. The model can be used as an estimator.

[0082] For example, the input of the first model is a question from a user, and the output of the first model is the answer to that question. If the output is an answer to the question, the second model, the evaluator, Is the output an answer? In other words, is the output valid as an answer to the question? At least one of these evaluations may be based on the content of the output. The correctness of the answer, or whether the output maintains the format (style) of an answer. In the case of natural language processing, the evaluator may include semantic analysis and sentence structure analysis. Each evaluation item may be evaluated by performing a pulse analysis.

[0083] Also, evaluating the validity, validity, or appropriateness of the generated answers means The answer itself can be evaluated, or the first model that generated the answer can be evaluated using the search results. The output from the first model (data 3) and the output from the first model not augmented with the search results (data 4) Conceptually, this also includes evaluating by comparing the

[0084] At least one processor in the information processing system may, for example, provide, using the second model, The probability distribution P of the third data generated using the provided data as search data and the probability distribution P of the third data generated using the provided data as search data are Using the probability distribution P' of the fourth data generated without running the program, we set the expected value as shown below. The contribution can be calculated using the above.

number

[0085] Here, C(S) is the output value of the evaluator for sample S, and E(·) denotes the expected value. The information processing system calculates the expected value of C for sample S obtained according to probability distribution P and the probability distribution The expected value of C of sample S' according to distribution P' is used to calculate the probability of the sample being broken down into two parts. One or more samples are taken for the rate variables S~P(S) and S'~P'(S') The expected values ​​of C(S) and C(S') are calculated using the average of

[0086] In this way, the information processing system samples based on the probability distributions P and P' and calculates the evaluation value By calculating the expected value of , we can calculate the contribution of the second stage. In other words, The information processing system is divided into two models: the first model, which is augmented with search results, and the second model, which is not augmented with search results. A first model and a second model that evaluates the answers generated by the first model are used to generate contributions. The second model, which outputs the evaluation value, can be trained by any method. The evaluation value calculated by this second model is, as mentioned above, the more accurate the answer, the higher the score. Outputs a high value.

[0087] Therefore, according to formula (3), the information processing system uses the provided information for searching to generate The expected value of the evaluation in the probability distribution of the third data generated without performing a search is The difference between the expected value of the evaluation in the probability distribution of the data and is calculated as the contribution in the second stage. can.

[0088] That is, the more correct the answer is when the provided information is used in a search, the more likely it is that the second stage will be successful. The degree of contribution is a large value.

[0089] As described above, the information processing system according to the above embodiment circulates information provided to the data group. The extent to which the answers are valid, effective, and appropriate, and the degree to which correct answers can be obtained It is possible to calculate the contribution that reflects how effective the project is in achieving its objectives.

[0090] The information processing system uses the contributions of the first stage and the second stage together. Alternatively, only one of the contributions can be used. By setting the contribution level as above, the information provider can receive the contribution level in the first stage. In the second stage, the contribution of the This can encourage the provision of information that will improve the validity, validity, and relevance of the data provided. Cut.

[0091] The contribution calculated above is, by way of example and not of limitation, the Loyalty to the information provider. It can be used as an indicator to realize some kind of processing such as payback as a yardstick. The information processing system may, for example, provide royalties (monetary or otherwise) based on the degree of contribution of the information. The information provider may be given a royalty (commendation equivalent to the amount of the incentive) in return for the information provided. This concept includes incentives, bonuses, fees, points, etc. For example, the amount of royalties is linked to the identification code of the information provider. It may be done in a different way.

[0092] In the above, the information processing system obtains information from a data group by searching and uses the first model. If royalties are paid to providers of information used to generate answers to The amount of royalties will be determined according to the contribution of the information. , but is not limited to the above examples.

[0093] The information processing system, for example, retrieves data from a data group and processes the data according to a first model. The information used to generate the answer is not calculated based on the contribution of that information, but is instead based on the magnitude of its contribution. Instead, a flat royalty may be paid to the provider of the information.

[0094] Alternatively, the information processing system may, for example, retrieve the first model from the data set by searching. Calculate the contribution of information used to generate a response by the rule, and Even if a fixed royalty were paid to the information provider when the contribution of the information exceeds a certain level, good.

[0095] That is, the information processing system generates a question (first data) for which an answer is generated by the first model. ) and generate answers to the questions (third data) from the data set based on the questions. The information used for the search is obtained through searches, and royalties are paid to the providers of the information. It may be configured.

[0096] The information processing system generates answers using the first model and performs search using a search engine. The data can be divided into two categories: the system and the data used by the search engine. The management system uses one of these as an API (Application Programming Interface). It can also be provided.

[0097] For example, as a whole, we provide an API for the input and output of the first model, and the information provided by the information provider To provide a system for obtaining answers with search reinforcement using the collected information as a search target. This can be done.

[0098] For example, we provide APIs for input and output of search engines and use the information provided by information providers. It is possible to provide a system that uses such information as a search target.

[0099] For example, we provide APIs for accessing data sets and use them to search engines. The system can provide a path to access the data.

[0100] In this way, the system can be divided arbitrarily. In providing I, according to the information processing system of the above embodiment, the provider of the data It is possible to give an appropriate contribution level to each

[0101] That is, the information processing system is a first model, a search engine, or a data set The form may include at least one of at least a portion of the sub-data groups.

[0102] The information processing system generates third data as a plurality of different information processing devices. The information processing device may include an information processing device that calculates the contribution degree, and the same information The processing device may be configured to generate the third data and calculate the degree of contribution. That is, the information processing system includes one or more information processing devices. The processing of the present disclosure in the device is appropriately distributed to the one or more information processing devices. The processes may be performed individually or in a consolidated manner.

[0103] When the information processing system has the first model, the information processing device that sets the contribution degree is The second data output from the first model held by the system is searched for by a search engine inside or outside the information processing system. The third data is the output from the first model, which uses the data obtained by passing it to the engine. The contribution rate is calculated from the fourth data output from the database without performing a search. It is possible to assign an ID to each piece of data (information) in a data group and to use this ID (information) The contribution level is set for each information ID, and the data provided by the same provider is retrieved from the ID. Their contributions can also be summarised.

[0104] If the information processing system has a search engine, the information processing system that sets the contribution level is , based on the output from the first model, which is inside or outside the information processing system, The contribution level can be set by obtaining the third data. The results of using a search engine in the information processing system can be passed to the first model. do.

[0105] When an information processing system has at least a part of the data set, the information processing system sets the contribution degree. The information processing system is a system that processes the first Using the model and search engine, the third and fourth data are obtained and the contribution level is set. This can be added to each piece of provided information in the data set.

[0106] The data set does not need to be stored in its entirety, and some of it may be stored in secret and encrypted. For example, a company may provide an API to access the collected keys. Each company stores confidential data in secret and requires a The present invention may be in a form in which paths and interfaces for such communication are provided.

[0107] In this case, the sub-data groups that exist in each company must be concealed from the outside. Therefore, users who want to use the first model can choose which company's data to use. On the other hand, the information processing system cannot detect whether a user is using a search based on the search term. By referencing the ID (information ID) linked to the data in the data group, you can identify which subdata It is possible to understand whether the answer is based on information obtained from the group.

[0108] The information processing system uses the information (search results) used to generate the answer by the first model. Information identifying the provider of the information obtained through the and / or a string indicating the source or source of information, such as a URL, The user receiving the answer can identify that the answer was generated by referring to the information provided by the provider. The information may be output in such a manner that

[0109] The information processing system also processes the information used to generate the answer by the first model (the information obtained by the search). Information identifying the provider of the information obtained through the , and / or a character string indicating the source or source of information, such as a URL, etc.) You received a response stating that the response was generated using information you provided. The information must be output in a manner that allows the user (the recipient of the response) to identify it (for example, by outputting it together with the response). The information indicating the provider may be text, an image, or the like.

[0110] The information processing system uses the subdata that are the source of information acquired during actual operation. The contribution of each item to the information provider is evaluated based on the number of times and frequency of use of the item as an answer. It is also possible to grant it.

[0111] In either case, the first model can have a variety of forms. As explained above, a large-scale language model is a system in which text such as sentences or words is input and sentences are processed. The first model may be a model that outputs text such as a chapter or a word, etc. The model outputs an image or video when text is input, A model that outputs text, a model that inputs an image or video when an image or video is input, A model that outputs graph information when graph information is input, and a model that outputs text when graph information is input Or a model that outputs images or videos, and when text, images or videos are input, graph information is generated. Of course, the first model may be a model other than these. It's fine.

[0112] The trained model in the above embodiment may be, for example, trained as described above and then In addition, the concept may include models distilled using general techniques.

[0113] In the following description, each device in the embodiment refers to at least one device that exists in the information processing system. The device may be, for example, an information processing system that processes information on a data set. A device for setting a contribution degree at the timing when the information is provided, or a device for generating third data Alternatively, the device may be a device that uses the first model, or or a device that utilizes a search engine, or at least one of these. Alternatively, the present invention may be an apparatus that performs a plurality of operations in combination.

[0114] Some of the devices (information processing devices provided in the information processing system) in the above-described embodiments Or it may be entirely composed of hardware, such as a CPU (Central Processing Unit) or information on the software (programs) executed by GPUs (Graphics Processing Units), etc. In the case where the information processing is performed by software, The software that realizes at least a part of the functions of each device in the form of Non-transitory storage media such as Disc-Read Only Memory, Universal Serial Bus (USB) memory, etc. By storing the information in a non-transitory computer-readable medium and reading it into a computer, The software may be used to execute information processing. Furthermore, the whole or part of the processing of the software may be ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate When the software is implemented in a circuit such as a PLC, the information processing is performed by the hardware. The processing may be performed by software.

[0115] The storage medium for storing the software may be a removable medium such as an optical disk, or a hard disk. The storage medium may be a fixed storage medium such as a hard disk or memory. It may be installed inside the computer (such as a main memory or auxiliary memory), or it may be installed outside the computer. The section may be provided with:

[0116] FIG. 5 shows each device in the above-described embodiment (information processing device provided in the information processing system). FIG. 1 is a block diagram showing an example of a hardware configuration of a processor. A processor 71, a main memory device 72 (memory), an auxiliary memory device 73 (memory), and a network The device interface 75 is connected to the bus 76. The present invention may be realized as a computer 7 connected in a manner similar to that of the first embodiment.

[0117] The computer 7 in FIG. 5 has one of each component, but it may have multiple of the same components. In addition, in FIG. 5, one computer 7 is shown, but the software may be It is installed on multiple computers, and each of the multiple computers uses the software. In this case, the computer may execute the same or different parts of the same software. Distributed computers in which each communicates via a network interface 74 and executes processing. In other words, each of the devices (information processing devices) in the above-described embodiment may be in the form of a single routing. An information processing device provided in the system) executes instructions stored in one or more storage devices on one or more may be configured as a system that realizes functions by being executed by multiple computers. In addition, the information transmitted from the terminal is transmitted to one or more computers in the cloud. The data may be processed by the data processor and the processing result may be transmitted to the terminal.

[0118] Each of the devices (information processing devices provided in the information processing system) in the above-mentioned embodiment The seed calculation may be performed using one or more processors, or multiple computers via a network. The operations may be executed in parallel using a computer. The processing may be distributed to a number of computing cores and executed in parallel. Some or all of the means are cloud services that can communicate with computers 7 via a network. The present invention may be implemented by at least one of the processor and the memory device. As described above, each device in the above-described embodiment is a parallel computer using one or more computers. It may be a form of computing.

[0119] The processor 71 is an electronic circuit that performs at least one of the control and calculation functions of the computer. (Processing circuit, Processing circuitry, CPU, GPU, FPGA, ASIC, etc.) The processor 71 may be a general-purpose processor, or a processor that is programmed to perform a specific operation. A semiconductor device that includes a specially designed processing circuit or both a general-purpose processor and a specially designed processing circuit. The processor 71 may include an optical circuit. Alternatively, it may include a computing function based on quantum computing.

[0120] The processor 71 processes data and software input from each device in the computer 7. The calculation results and control signals may be output to each device, etc. The processor 71 may be a processor for executing an OS (Operating System) of the computer 7 and an application. The components constituting the computer 7 may be controlled by executing the operations. stomach.

[0121] Each device (information processing device provided in the information processing system) in the above-mentioned embodiment is The present invention may be implemented by one or more processors 71. Here, the processor 71 may be a single chip. It may refer to one or more electronic circuits arranged on a chip, or it may refer to two or more chips or two The term may refer to one or more electronic circuits disposed on the above device. When used, the electronic circuits may communicate by wires or wirelessly.

[0122] The main memory 72 may store instructions executed by the processor 71 and various data. The information stored in the main memory 72 may be read by the processor 71. The memory device 73 is a storage device other than the main memory device 72. It means any electronic component capable of storing information, including semiconductor memory. The memory may be either a volatile memory or a non-volatile memory. Each device (information processing device provided in the information processing system) stores various data, etc. The storage device for performing the above may be realized by a main storage device 72 or an auxiliary storage device 73. The above-mentioned embodiment may be realized by an internal memory of the processor 71. The storage unit of the information processing device provided in the information processing system in the embodiment is a main storage device 72 or may be realized by the auxiliary storage device 73.

[0123] Each device in the above-described embodiment (information processing device provided in the information processing system) At least one storage device (memory) and a At least one processor that supports the In addition, at least one memory may be connected to each processor. A storage device may be connected to at least one of the plurality of processors. may include a configuration in which the storage device is connected to at least one of the plurality of storage devices. This configuration is also realized by the storage devices and processors contained in the multiple computers. In addition, the memory device may be integrated with the processor (e.g., the L1 cache). The L2 cache may include a cache memory including a L3 cache.

[0124] The network interface 74 is connected to the communication network 8 by wireless or wired means. The network interface 74 is an interface for connecting to existing communication standards. Network Interface The sensor 74 exchanges information with an external device 9A connected via a communication network 8. The communication network 8 may be a WAN (Wide Area Network), a LAN (Local Area Network), Area Network), PAN (Personal Area Network), or a combination of these. The computer 7 and the external device 9A may be connected to each other, and information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, and an example of a LAN is the IEEE 802. Examples of PAN include Bluetooth and NFC. (Near Field Communication), etc.

[0125] The device interface 75 is an interface such as a USB that directly connects to an external device 9B. It is.

[0126] The external device 9A is a device connected to the computer 7 via a network. External device 9B is a device that is directly connected to computer 7.

[0127] The external device 9A or the external device 9B may be, for example, an input device. For example, cameras, microphones, motion capture, various sensors, keyboards, etc. , a mouse or a touch panel, and gives the acquired information to a computer 7 In addition, the input and output units of personal computers, tablet terminals, smartphones, etc. The device may be a device having a memory and a processor.

[0128] In addition, the external device 9A or the external device 9B may be, for example, an output device. For example, LCD (Liquid Crystal Display), organic EL (Electro Luminescence) panels, etc. It may be a display device for outputting a sound or a speaker for outputting a sound. A computer, tablet, or smartphone with an output section, memory, and processor The device may be equipped with a

[0129] In addition, the external device 9A or the external device 9B may be a storage device (memory). For example, The external device 9A may be a network storage device, and the external device 9B may be a hard disk drive (HDD), etc. The storage may be

[0130] The external device 9A or the external device 9B may be any of the devices (information processing devices) in the above-described embodiments. It may be a device having some of the functions of the components of an information processing device provided in the system. That is, the computer 7 transmits a part or all of the processing result to the external device 9A or the external device 9B. It may transmit the processing result or receive a part or all of the processing result from the external device 9A or the external device 9B. This is also fine.

[0131] In this specification (including the claims), "at least one of a, b, and c" or "a When the expression "at least one of (a), b or c" (including similar expressions) is used, , b, c, a - b, a - c, b - c or a - b - c. Also, a - a, a - b - Contains multiple instances of any element, such as a - b, a - a - b - b - c - c, etc. Furthermore, a list of elements (a, b and c) may be used, such as a - b - c - d, with d. ) This also includes adding other elements other than the above.

[0132] In this specification (including the claims), the term "data as input / using / based on data" is used. When expressions such as "in accordance with" or "according to" (including similar expressions) are used, In cases where there is no data, the data itself may be used, or data that has been processed in some way (e.g. For example, noise-added data, normalized data, features extracted from data, intermediate tables of data, This also includes cases where data is used as input, using data, or based on data. In cases where it is stated that a result will be obtained "in accordance with" or "in accordance with" (or similar expressions are used), Unless otherwise specified, the results may be obtained based solely on the data in question, The results may be affected by other data, factors, conditions and / or circumstances other than the data. Also, when it is stated that "data will be output" (or when a similar expression is used), Unless otherwise specified, the data itself may not be used as output or may have no effect on the data. Some processing (e.g., noise addition, normalization, extraction from data) This includes cases where the output is a representation of the features obtained by the machine learning or intermediate representation of data.

[0133] In this specification (including the claims), the terms "connected" and "coupled" are used interchangeably. When the term "coupled" is used, it means a direct connection / coupling, an indirect connection / coupling, Electrically connected / coupled, communicatively connected / coupled, function This includes both operatively and physically connected / coupled. The term is intended as an open-ended term that includes but is not limited to the following: Connections / couplings that should be interpreted appropriately but are not intentionally or naturally excluded should be construed without limitation as being included in the term.

[0134] In this specification (including the claims), "A configured to B" " is used, the physical structure of element A has a configuration capable of performing operation B. and the permanent or temporary setting of element A. The configuration is configured / set to actually perform action B. For example, if element A is a general-purpose processor, the processor may perform operation B. and have a hardware configuration capable of executing the above, and By setting a program (instruction) that is symmetrical, the action B is actually executed. In addition, if element A is a dedicated processor or dedicated arithmetic circuit, etc. Regardless of whether or not the control instructions and data are actually attached, the processing of the processor It is sufficient that the path structure, etc. is implemented to actually perform action B.

[0135] In this specification (including the claims), terms that imply containing or possessing (e.g., "includes ( Where the words "comprising / including" and "having" are used, open-ended, including when it contains or possesses something other than the object indicated by the object of the word These terms are intended as terms of quantity. If the expression is not singular or suggests a singular number (expressions using the articles a or an), the expression is It should not be construed as being limited to any particular number.

[0136] In certain places in this specification (including the claims), the words "one or more " or "at least one" and other expressions such as "quantity" are used elsewhere. The term "a" or "an" is used without specifying the number or implying a singular number. However, the latter expression is not intended to mean "one." Expressions suggesting a single or singular number (such as the articles a or an) do not necessarily limit the number to a particular number. should not be construed as being

[0137] In this specification, certain configurations of certain embodiments may have certain advantages. If it is stated that the above-mentioned result can be obtained, unless there is a special reason, It should be understood that the same advantages may also be achieved in one or more other embodiments. However, the presence or absence of such effects generally depends on various factors, conditions and / or states, etc. It should be understood that the above effect is not necessarily achieved by the above. The present invention relates to a method for obtaining a function, ... In the invention according to the claim which defines the said structure or a similar structure, This effect is not necessarily achieved.

[0138] In this specification (including the claims), the term "maximize" is used interchangeably with "maximize" or "maximize When terms such as "maximum" are used, it is important to find the global maximum. Finding the maximum approximation, finding the local maximum and approximating the local maximum The terms should be interpreted appropriately according to the context in which they are used, including determining the value of the terms. It also includes finding approximations of these maximum values ​​probabilistically or heuristically. When terms such as "minimize" or "minimization" are used, ,Finding the global minimum,Finding an approximation to the global minimum,Local This term includes finding the global minimum and approximating the local minimum. The minimum values ​​should be interpreted appropriately according to the context in which they are used. Similarly, "optimize" includes "optimizing" a problem by using a specific method or heuristics. When terms such as "optimization" are used, it means to find the global optimum. , finding an approximation of the global optimum, finding a local optimum, and This includes determining an approximation of the optimal value for a given parameter, and should be interpreted accordingly depending on the context in which the term is used. In addition, these optimal values ​​should be approximated probabilistically or heuristically. This includes:

[0139] In this specification (including the claims), when multiple pieces of hardware perform a given process, The hardware may cooperate to perform a predetermined process, or a part of the hardware may perform a predetermined process. Alternatively, some hardware may perform a part of the predetermined processing, and other hardware may perform the other processing. The remaining processing may be performed by the hardware. Or, a plurality of pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process. When expressions such as "processing" (including similar expressions) are used, the hardware that performs the first processing The hardware for performing the first and second processes may be the same or different. That is, the hardware performing the first processing and the hardware performing the second processing are the one or more The hardware may be included in the number of pieces of hardware. It may also include devices including circuits.

[0140] In this specification (including the claims), a plurality of storage devices (memories) are used for storing data. In this case, each of the multiple storage devices may store only a portion of the data, The entire data may be stored. Alternatively, some of the multiple storage devices may store the data. It may also include a configuration for storing the information.

[0141] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-described individual embodiments. The present invention is not intended to be limited to the above-mentioned embodiments, but rather to be construed as being in accordance with the principles of the present invention. Various additions, modifications, substitutions, and other modifications may be made without departing from the scope of the conceptual idea and spirit of this disclosure. For example, in the above embodiment, the numerical values ​​or formulas are explained. When used herein, they are provided for illustrative purposes only and are not intended to limit the scope of the disclosure. In addition, the order of operations shown in the embodiment is merely an example, and is not intended to be construed as limiting the scope of the present disclosure. This does not limit the scope of the present invention.

Claims

1. one or more memories; one or more processors; Equipped with The one or more processors: Obtain a question for which an answer is generated by the first model; The first model is used to generate an answer to the question from a set of data based on the question. obtain information used by calculating a contribution of the information to generating the answer by the first model; Information processing device.

2. The one or more processors may pay royalties in an amount according to the degree of contribution of the information, to the provider of said information; The information processing device according to claim 1.

3. one or more memories; one or more processors; Equipped with The one or more processors: Obtain a question for which an answer is generated by the first model; The first model is used to generate an answer to the question from a set of data based on the question. obtain information used by granting royalties to the provider of said information; Information processing device.

4. The one or more processors may further include a processor for determining whether the first model is generating the answer based on the information. the royalty is awarded to the information provider in accordance with the degree of contribution of the information provider to the information provider; The information processing device according to claim 3.

5. The one or more processors: information indicating a provider of the information used in generating the answer by the first model; The recipient of the answer can identify that the answer was generated using information provided by the provider. Output in the form possible, 5. An information processing device according to claim 1.

6. The one or more processors generate the first model augmented with the information and calculating the contribution using the first model, which is not augmented with information; 5. An information processing device according to claim 1, 2 or 4.

7. The first model is a probabilistic model.

5. An information processing device according to claim 1, 2 or 4.

8. the first model is a language model; 5. An information processing device according to claim 1, 2 or 4.

9. The one or more processors: using the information to generate an answer to the question using the first model; 5. An information processing device according to claim 1, 2 or 4.

10. The one or more processors: Retrieving information used by the first model in generating an answer to the question The second data is obtained for The second data is input to a search engine and a search is performed to find a desired item from the data set. Obtaining said information; 5. An information processing device according to claim 1, 2 or 4.

11. the second data is data obtained by inputting the question into the first model; The answer is information obtained using the search engine based on the second data, Data generated by inputting the first model, The information processing device according to claim 10.

12. The one or more memories include not storing the first model; The information processing device according to claim 1.

13. The one or more memories include storing the first model; The information processing device according to claim 1.

14. The one or more processors include at least Data generated using the first model augmented with the information is If the data is more valid than data generated using the first model that is not enhanced, Calculating the degree of contribution so that it is higher; The information processing device according to claim 2.

15. The one or more processors include at least A distribution of data generated using the first model augmented with the information, and A distribution of data generated using the first model that is not augmented with information, , calculating the contribution to the provided information; 3. The information processing device according to claim 2.

16. The one or more processors: Calculating the contribution of the information for each provider of the provided information; Information according to any one of claims 1 to 4 or any one of claims 12 to 15 Processing unit.

17. At least a part of the data in the data group is kept confidential by the provider of the data. Managed by The information processing device according to claim 16.

18. One or more information processing devices are provided, The one or more information processing devices include Obtain a question for which an answer is generated by the first model; The first model is used to generate an answer to the question from a set of data based on the question. obtain information used by Calculating a contribution of the information to generating the answer by the first model; one or more of the information processing devices that generate answers to the questions; The one or more information processing devices that calculate the degree are other information processing devices or at least At least a part of the information processing devices are the same. Information processing system.

Images