Information processing system, information processing device, information processing method, and program

Abstract

To provide a technique for generating output information for multiple pieces of information by using minimal computational resources.SOLUTION: An information processing system is configured to: obtain information related to an output candidate and a plurality of pieces of target information; calculate first intermediate data by inputting the information related the output candidate into a machine learning model; and generate output information for each of the plurality of pieces of the target information by executing single inference processing using the machine learning model for each of the plurality of pieces of the target information by using at least a part of the first intermediate data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing device, an information processing method, and a program. [Background technology]

[0002] Machine learning models such as large language models (LLMs) are well known. Large language models generate output information for each predetermined processing unit, such as a token, so techniques have been proposed to efficiently handle large inputs and outputs. For example, there is a technique called key-value cache, which caches data calculated by a large language model during decoding. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Omri Mallis, "Techniques for KV Cache Optimization in Large Language Models", [online], [Retrieved May 2, 2024], Internet<URL: https: / / www.omrimallis.com / posts / techniques-for-kv-cache-optimization / > Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide a technology for generating output information for each of a plurality of pieces of information with few computational resources. [Means for solving the problem]

[0005] An information processing system according to one aspect of the present disclosure includes at least one memory and at least one processor, wherein the at least one processor acquires information regarding output candidates and multiple pieces of target information, calculates first intermediate data by inputting the information regarding the output candidates into a machine learning model, and generates output information for each of the multiple pieces of target information by using at least a portion of the first intermediate data to perform a single inference process using the machine learning model for each of the multiple pieces of target information. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a block diagram showing an example of the overall configuration of an information processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of the functional configuration of an inference device according to a first embodiment. FIG. [Figure 3] FIG. 4 is a diagram illustrating an example of an inference request according to the first embodiment. [Figure 4] FIG. 10 is a diagram for explaining an example of an inference process according to a comparative example. [Figure 5] FIG. 3 is a diagram illustrating an example of an inference process according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing an example of an input screen according to the first embodiment. [Figure 7] FIG. 3 is a diagram showing an example of an output screen according to the first embodiment. [Figure 8] 5 is a flowchart showing an example of an inference process according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing a first example of a prompt according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing a second example of a prompt according to the second embodiment. [Figure 11] FIG. 10 is a block diagram showing an example of the overall configuration of an information processing system according to a third embodiment. [Figure 12] FIG. 11 is a block diagram illustrating an example of the functional configuration of a generating device according to a third embodiment. [Figure 13] FIG. 11 is a diagram showing a first example of a prompt according to the third embodiment. [Figure 14] FIG. 11 is a diagram showing a second example of a prompt according to the third embodiment. [Figure 15] 11 is a flowchart showing an example of a generation process according to the third embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of an attention mask. [Figure 17] FIG. 13 is a diagram showing an example of an input screen according to the fourth embodiment. [Figure 18] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] [First embodiment] A first embodiment of the present disclosure is an information processing system that executes a predetermined task based on a machine learning model. The machine learning model according to this embodiment may be an autoregressive model. As an example, the autoregressive model may be a decoder-only large language model (LLM). The machine learning model may be, for example, a generative model, a base model, or a neural network that generates various types of data such as audio, images, and videos. The machine learning model may be multimodal.

[0009] The information processing system according to this embodiment executes a generation task for generating output information for information to be processed (hereinafter also referred to as "target information"). As an example, the generation task may be a classification task for classifying each of a plurality of target information into predetermined options.

[0010] The predetermined option may be expressed by a data length that can be generated in one inference process. The data length that can be generated in one inference process may be the maximum data length that can be generated by a neural network included in the machine learning model executing one inference process. As an example, if the machine learning model is a large-scale language model, the predetermined option may be expressed by one token. Note that a token is a processing unit when the machine learning model processes electronic data, and the amount of data per token may vary depending on the design of the machine learning model. As an example, a token is one Japanese character or one English word, but depending on the frequency of occurrence, one character may be represented by two tokens, or two or more characters may be represented by one token.

[0011] For example, the generation task according to this embodiment may be a task of assigning a probability of success for each of a plurality of options to each of a plurality of sentences, which are examples of target information. The sentences may include one or more sentences. For example, the sentences may be messages posted to a social networking service. For example, the options may be classifications of impressions that the sentences give. For example, the options for impression classifications may include "good impression," "bad impression," or "neither." For example, the probability of success for each option may be the probability of success for each option. In other words, the generation task may be a task of determining the degree of good or bad impression a message posted to a social networking service gives. Specifically, the generation task may be a task of generating output information indicating that the probability of a good impression is 0.7 and the probability of a bad impression is 0.3 for a certain post. The generation task is not limited to the above example and may be any task that generates output information of a predetermined length or less for each of a plurality of pieces of target information.

[0012] In the past, tasks that required classifying multiple pieces of target information into predetermined options required retraining of machine learning models for each classification task. Large-scale language models can perform a variety of tasks by training on large datasets, so they can perform any classification task without retraining. However, when performing classification tasks for multiple pieces of target information, inference processing must be performed using prompts with options for each piece of target information. As the number of pieces of target information increases, the overall required computing resources increase.

[0013] This embodiment provides a technology for generating output information for each of a plurality of pieces of information with minimal computational resources. In this embodiment, information about output candidates is input into a machine learning model to calculate intermediate data for the machine learning model, and the calculated intermediate data is used to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information. In one aspect, this embodiment caches and shares the intermediate data calculated using information about the output candidates when generating output information for each of the plurality of pieces of target information, thereby enabling output information for each of the plurality of pieces of target information to be generated with minimal computational effort. In another aspect, this embodiment eliminates the need to cache the calculated intermediate data when generating output information for each of the plurality of pieces of target information, thereby enabling output information for each of the plurality of pieces of target information to be generated with minimal memory usage.

[0014] <Overall configuration of information processing system> The overall configuration of an information processing system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of an information processing system according to the first embodiment.

[0015] 1, information processing system 1000 includes an inference device 10 and a terminal device 50. Inference device 10 and terminal device 50 may be connected to each other so as to be able to communicate data with each other via a communication network such as a LAN (Local Area Network) or the Internet.

[0016] The inference device 10 is an example of an information processing device such as a personal computer, a workstation, or a server that executes a predetermined task in response to an inference request from a terminal device 50. The inference device 10 may receive an inference request from the terminal device 50. The inference device 10 may transmit an inference result in response to the inference request to the terminal device 50.

[0017] The inference request is information or a signal that requests the execution of a predetermined task. In this embodiment, the predetermined task may be a task that generates a classification result in which each of a plurality of pieces of target information is classified into predetermined options.

[0018] The inference device 10 includes a machine learning model M. The machine learning model M is a machine learning model used to perform a predetermined task. The machine learning model M may be an autoregressive model, a generative model, a foundational model, or a neural network. As an example, the machine learning model M may be a decoder-only large-scale language model.

[0019] The machine learning model M may be realized by a single machine learning model. The machine learning model M may be realized by multiple machine learning models working together. The machine learning model M may be composed of multiple machine learning models according to the tasks to be performed. The machine learning model M may be built into an information processing device other than the inference device 10 (for example, a terminal device 50, another information processing device, etc.). The machine learning model M may be distributed and held in an external information processing system consisting of multiple information processing devices.

[0020] The inference device 10 may be realized in multiple information processing devices or information processing systems equipped with different machine learning models M. The inference device 10 may be realized in a single information processing device or information processing system equipped with multiple machine learning models M. The inference device 10 may execute a predetermined task using an external machine learning model M. Here, "external" means not included in the information processing system 1000.

[0021] The terminal device 50 is an example of an information processing device such as a personal computer, smartphone, or tablet terminal operated by a user of the information processing system 1000. The terminal device 50 may transmit an inference request to the inference device 10. The terminal device 50 may receive an inference result from the inference device 10 and present it to the user.

[0022] For example, the terminal device 50 may display the inference result on a display device of the terminal device 50. For example, the terminal device 50 may output a voice synthesized from the inference result from a speaker of the terminal device 50.

[0023] Presenting information to the user may include the processor executing at least a portion of the processing required to display the information on the display device. The display device may be provided in the same device as the processor or in a different device from the processor. The display device may be a plurality of display devices.

[0024] The overall configuration of the information processing system 1000 shown in FIG. 1 is an example, and various system configuration examples are possible depending on the application and purpose. The information processing system 1000 may be configured with one or more information processing devices. The information processing devices included in the information processing system 1000 may be a system configured with multiple devices. Each function included in the information processing system 1000 may be realized by any device that constitutes the system. Each component included in the information processing system 1000 may be included in any device that constitutes the system.

[0025] One or more of the inference device 10 and terminal device 50 may be included in the information processing system 1000. The inference device 10 may be implemented by multiple computers, or may be implemented as a cloud computing service. The division of devices such as the inference device 10 and terminal device 50 shown in Figure 1 is one example.

[0026] As an example, the information processing system 1000 may be configured with one or more server devices and one or more terminal devices 50. The one or more server devices may have one or more of the functions of the inference device 10. The server device may be realized as a system including multiple information processing devices. The server device may be realized as a cloud computing service.

[0027] As another example, the information processing system 1000 may be configured with a single information processing device. The information processing device may have the functions of the inference device 10 and the terminal device 50.

[0028] <Functional configuration of the inference device> The functional configuration of the inference device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the inference device according to the first embodiment.

[0029] 2, the inference device 10 includes a model storage unit 101, a state storage unit 102, a request receiving unit 110, a first inference unit 120, a second inference unit 130, and an output unit 140. The inference device 10 functions as the model storage unit 101, the state storage unit 102, the request receiving unit 110, the first inference unit 120, the second inference unit 130, and the output unit 140 by executing a pre-installed inference program by at least one processor.

[0030] The model storage unit 101 stores a machine learning model M in advance. The machine learning model M is trained in advance based on predetermined training data. For example, the machine learning model M may be trained by the inference device 10, or may be trained by an external information processing device or information processing system. The model storage unit 101 may store multiple machine learning models M.

[0031] The state storage unit 102 stores intermediate data of the machine learning model M when information about output candidates is input. The intermediate data stored in the state storage unit 102 is generated by the first inference unit 120.

[0032] The state storage unit 102 may be configured from memory of a GPU (Graphics Processing Unit) provided in the inference device 10. The state storage unit 102 may be configured from memory of a CPU (Central Processing Unit) provided in the inference device 10. The state storage unit 102 may be configured from an auxiliary storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) provided in the inference device 10.

[0033] The request receiving unit 110 receives an inference request. The request receiving unit 110 may receive an inference request from the terminal device 50. The inference request may be transmitted from the terminal device 50 in response to an operation on a screen displayed on the display device of the terminal device 50. The request receiving unit 110 may receive an inference request input to the inference device 10. The inference request may be input to the inference device 10 in response to an operation on a screen displayed on the display device of the inference device 10. At least a portion of the information included in the inference request may be generated by the inference device 10. The request receiving unit 110 may acquire an inference request from another information processing device.

[0034] In this embodiment, the inference request is information or a signal requesting the generation of output information for the target information. The inference request may include information regarding output candidates and target information. The information regarding the output candidates may include information regarding output candidates generated by an inference process using the machine learning model M. The information regarding the output candidates may further include information requesting an inference process using the machine learning model M. The target information may include information regarding the target of the inference process using the machine learning model M. The inference request may include multiple pieces of target information. The inference request may include only the target information, and the information regarding the output candidates may be obtained from the inference device 10 or another information processing device. Hereinafter, the information regarding the output candidates will also be referred to as "candidate information."

[0035] The candidate information may include options. The options may represent one or more items, elements, contents, etc. included in multiple classifications, multiple categories, multiple classes, multiple attributes, multiple groups, multiple types, multiple segments, multiple genres, multiple types, multiple divisions, multiple ranks, multiple grades, etc. The options may include option identification information. The option identification information may be represented using information (e.g., one token) that can be generated in a single inference process by the machine learning model M, and may be, for example, numbers, symbols, letters, etc. In the following example of candidate information, numbers and letters are used as option identification information. For example, the machine learning model M may be configured to represent option identification information such as "1", "a", and "A" each using one token. Example 1: Please choose the classification for the following text from the options below:

[0036] Select from: 1: Good impression 2: Bad impression 3: Neither Example 2: Please choose from the options below whether the following text is appropriate as information about Company A.

[0037] Select from: A: Suitable B: Not suitable c: Neither" Example 3: Please classify the news titles below into the following categories:

[0038] A. Politics B. Economy C. Entertainment D. Sports E.Science F.Others Example 4: Please choose the season that the following sentence describes from the options below.

[0039] Select from: 1: Spring 2: Summer 3: Autumn 4: Winter

[0040] The candidate information may include options. Here, the options may be information that can be represented using information (e.g., one token) that can be generated in one inference process by the machine learning model M. Examples of candidate information include the following. For example, the machine learning model M may be configured to represent the options "Spring", "Summer", "Fall", and "Winter" each using one token. Example 5: Please choose the season that the following sentence describes from the options below.

[0041] Select from: Spring Summer Fall Winter"

[0042] The candidate information may include information requesting the generation of a number. The number may represent a score, a mark, a numerical value, etc. The number may be represented using information (e.g., one token) that can be generated in one inference process by the machine learning model M. Examples of candidate information include the following. For example, the machine learning model M may be configured to represent each of the numbers from "0" to "9" using one token. Example 6: Please rate the following sentences on a scale of 0 to 9 to see how well they are written in business etiquette. Example 7: Please rate the following sentences out of 100:

[0043] The candidate information may include information requesting the generation of information (e.g., one token) that can be generated by the machine learning model M in one inference process. The information that is requested to be generated may be a question that can be answered using information that can be generated by the machine learning model M in one inference process. The information that is requested to be generated may include information that can identify options based on common sense or context. Examples of candidate information include the following. In this example, "Spring", "Summer", "Fall", and "Winter" can be identified as answer options, and the machine learning model M may be configured to represent "Spring", "Summer", "Fall", and "Winter" each with one token. Example 8: Please answer in English what season the following sentences refer to.

[0044] The candidate information may include reference answer information. The reference answer information may include example answers, response examples, reply examples, etc. to requests, questions, etc. The reference answer information may include specific target information and example answers, response examples, reply examples, etc. to the information. The reference answer information may include, as the example answers, response examples, reply examples, identification information of options that can be generated in a single inference process by the machine learning model M, options that can be generated in a single inference process by the machine learning model M, and information that specifies numbers that can be generated in a single inference process by the machine learning model M. For example, as reference answer information for the above example 1 of candidate information, the candidate information may include the following reference answer information: "Sentence If you have any questions or concerns, please feel free to contact us.

[0045] answer 1"

[0046] The target information may be a document or a character string. The target information may include text, an image, audio, or video. The target information may be a combination of two or more of text, an image, audio, or video. For example, when the candidate information includes options, the target information may be the information to be classified itself, or information that can identify the information to be classified.

[0047] The first inference unit 120 calculates intermediate data of the machine learning model M stored in the model storage unit 101 based on the candidate information included in the inference request received by the request receiving unit 110. The first inference unit 120 may calculate the intermediate data by inputting the candidate information to the machine learning model M. The first inference unit 120 may acquire the intermediate data output by the machine learning model M when the candidate information is input to the machine learning model M.

[0048] Note that inputting information into the machine learning model M may include directly or indirectly inputting the information into the machine learning model M. As an example, inputting information into the machine learning model M may include inputting the information directly into the machine learning model M. Furthermore, inputting information into the machine learning model M may include inputting other information generated based on the information into the machine learning model M.

[0049] The intermediate data of the machine learning model M may be at least a portion of the intermediate data calculated by the machine learning model M executing an inference process on the candidate information. The intermediate data may, for example, include information regarding the state of a hidden layer of a neural network included in the machine learning model M. The intermediate data may, for example, include information regarding the intermediate state of the machine learning model M. The intermediate data may, for example, be a cache used by the machine learning model M, or may be reusable information for reducing the amount of calculation in a subsequent inference process.

[0050] The intermediate data of the machine learning model M may be, for example, a key-value cache. That is, the intermediate data of the machine learning model M may include key vectors and value vectors out of the query vectors, key vectors, and value vectors used by the attention mechanism of the Transformer. The key-value cache may be an array of key vectors and value vectors for each token calculated when candidate information is input to the machine learning model M.

[0051] Here, caching may refer to the process of retaining or storing data obtained by calculation that will be reused in a storage device or the like, or the retained or stored data itself. Caching may also refer to the execution of the process of retaining or storing data obtained by calculation that will be reused in a storage device or the like. Data that will be reused may be retained or stored in a high-speed storage device. Key-value caching may include the process of storing key vectors and value vectors, or may include the stored key vectors and value vectors. Cached data may be erased at a predetermined timing (such as after reuse is completed).

[0052] The first inference unit 120 may generate model input information based on candidate information included in the inference request. The first inference unit 120 may input the model input information to the machine learning model M. The first inference unit 120 may execute one inference process using the machine learning model M by inputting the model input information to the machine learning model M, and obtain intermediate data calculated by the inference process.

[0053] The inference process using the machine learning model M may indicate a process that does not directly aim to obtain an inference result (in other words, output information from the machine learning model M). The inference process performed by the first inference unit 120 is used to calculate intermediate data, and therefore the inference result obtained by the inference process may be discarded. Furthermore, the inference process performed by the first inference unit 120 only needs to include a process for calculating intermediate data using candidate information, and may not include any processing after the calculation of the intermediate data. The inference process using the machine learning model M may, for example, include forward processing using the machine learning model M.

[0054] The first inference unit 120 may use the candidate information itself as the model input information. The first inference unit 120 may generate the model input information by processing the candidate information. The first inference unit 120 may generate the model input information based on part of the candidate information. The first inference unit 120 may generate the model input information by extracting part of the information from the candidate information.

[0055] The first inference unit 120 may generate the model input information based on the machine learning model M. The first inference unit 120 may generate the model input information based on another machine learning model (e.g., a generative model, a base model, a neural network, etc.). The other machine learning model may be stored in the model storage unit 101. The other machine learning model may be stored in an external information processing device or information processing system.

[0056] The first inference unit 120 may generate model input information by embedding candidate information in a predetermined template. The template may include one or more placeholders for embedding candidate information. The template may include instruction information that instructs execution of a task. The template may include constraint information related to the model input information. The instruction information and constraint information may be predetermined fixed sentences.

[0057] The template may be optimized by any prompt tuning method to obtain good output information. For example, the template may be optimized by searching for an optimal template using a genetic algorithm or the like based on a benchmark that evaluates prompts generated using the template.

[0058] The model input information may include text data, image data, or audio data. The text data may be, for example, a natural language sentence called a prompt. The image data may be, for example, a still image or a video. The text data may be text data obtained by speech recognition of audio data or audio data recorded in a video. The text data may be text obtained by character recognition of image data. The audio data may be audio data.

[0059] The first inference unit 120 stores the intermediate data of the machine learning model M when the candidate information is input in the state storage unit 102. The first inference unit 120 may store the intermediate data itself. The first inference unit 120 may store information that can identify the intermediate data. If the first inference unit 120 stores information that can identify the intermediate data, when using the intermediate data, the first inference unit 120 may identify the intermediate data based on the identifiable information and read out the identified intermediate data.

[0060] The first inference unit 120 may store the intermediate data in a storage device other than the state storage unit 102 or an information processing device other than the inference device 10. The first inference unit 120 may transmit the intermediate data to the terminal device 50, thereby causing the terminal device 50 to store the intermediate data. The first inference unit 120 may store the intermediate data in an external storage device or information processing device.

[0061] The first inference unit 120 may store the machine learning model M, the intermediate data, and the candidate information in association with each other. At least one of the machine learning model M, the intermediate data, and the candidate information may be associated with each other using identification information that identifies each of them. The first inference unit 120 may associate the machine learning model M, the intermediate data, and the candidate information with each other using any method that allows one piece of information to be identified from another piece of information.

[0062] Note that storing the information may include at least one processor performing at least a portion of the processing required to store the information. Storing the information may include caching the information.

[0063] If intermediate data corresponding to the same candidate information is already stored in the state storage unit 102, the first inference unit 120 does not need to calculate intermediate data corresponding to the candidate information. As an example, when executing a routine task in which the same task is repeatedly executed periodically or irregularly, previously generated intermediate data may be used as is. The first inference unit 120 does not need to recalculate the intermediate data generated in the first task for the second or subsequent tasks, thereby reducing the computational resources required to execute the task.

[0064] The first inference unit 120 may specify whether or not to use a cache when causing the machine learning model M to execute an inference process. Using a cache may include caching key vectors and value vectors calculated when the machine learning model M executes forward processing. The first inference unit 120 may specify that the key vectors and value vectors be cached.

[0065] The second inference unit 130 uses the intermediate data calculated by the first inference unit 120 to execute a single inference process using the machine learning model M stored in the model storage unit 101 for each of the plurality of pieces of target information included in the inference request received by the request receiving unit 110, thereby generating output information for each of the plurality of pieces of target information. The second inference unit 130 may read the intermediate data stored (cached) in the state storage unit 102 and input one piece of target information and the intermediate data to the machine learning model M, thereby generating output information for the target information. The second inference unit 130 may acquire the output information output by the machine learning model M when the target information and the intermediate data are input to the machine learning model M. When executing the inference process using the intermediate data, the second inference unit 130 may read the cached intermediate data corresponding to each layer included in the machine learning model M from the state storage unit 102 and use the cached intermediate data corresponding to the layer in the calculation in the layer.

[0066] The second inference unit 130 may generate output information in parallel for two or more pieces of target information among the multiple pieces of target information. Hereinafter, parallel processing of two or more pieces of target information is also referred to as "batch processing," and the number of pieces of target information to be processed in parallel is also referred to as "batch size." The second inference unit 130 may cause one GPU included in the inference device 10 to batch process the target information. The second inference unit 130 may cause multiple GPUs included in the inference device 10 to batch process two or more pieces of target information. Note that the first inference unit 120 may calculate intermediate data using candidate information and store the intermediate data in the state storage unit 102 on a batch basis.

[0067] When batch processing multiple pieces of target information, the second inference unit 130 may copy the intermediate data cached in the state storage unit 102 by the batch size and have the GPU process the copies. The second inference unit 130 may copy the intermediate data only when necessary. As an example, in forward processing using a machine learning model M having multiple layers, the second inference unit 130 may copy only the information required for each layer of the intermediate data.

[0068] The second inference unit 130 may generate model input information based on target information included in the inference request. The second inference unit 130 may generate multiple pieces of model input information based on each of multiple pieces of target information. The second inference unit 130 may divide the target information into multiple chunks and generate model input information for each chunk. The second inference unit 130 may input the model input information and intermediate data to the machine learning model M.

[0069] The second inference unit 130 may execute a single inference process using the machine learning model M by inputting the model input information and intermediate data to the machine learning model M, and may obtain output information generated by the inference process. The second inference unit 130 may recalculate part of the intermediate data during the inference process.

[0070] The second inference unit 130 may not store intermediate data of the machine learning model M calculated when the target information is input. The second inference unit 130 may not store intermediate data calculated in the course of executing one inference process on the target information, for use in inference processes subsequent to one inference process using the machine learning model M. The second inference unit 130 may be configured not to store intermediate data calculated in the course of executing one inference process on some of the target information.

[0071] Note that not storing intermediate data for inference processing after one inference processing using the machine learning model M may include at least any of not caching the intermediate data for the subsequent inference processing, not executing an instruction to cache the intermediate data for the subsequent inference processing, deleting the intermediate data before executing the subsequent inference processing, and executing an instruction to release the intermediate data before executing the subsequent inference processing. Not storing intermediate data for inference processing after one inference processing using the machine learning model M does not have to be based on the assumption that the subsequent inference processing will be executed. In other words, it may be the case that a setting is made such that the subsequent inference processing will not be executed, and only one inference processing is executed.

[0072] The second inference unit 130 may specify whether or not to use a cache when causing the machine learning model M to execute an inference process. The second inference unit 130 may specify that the key vectors and value vectors cached in the state storage unit 102 are to be referenced. The second inference unit 130 may specify that the key vectors and value vectors calculated by executing an inference process on the target information are not to be cached. If the second inference unit 130 does not cache the key vectors and value vectors, the memory size used for the key-value cache can be reduced. As a result, the batch size in batch processing can be increased, thereby achieving more efficient batch processing.

[0073] The second inference unit 130 may add a fixed token to the end of the target information. The fixed token may be a token that indicates the end of the target information, or a token that indicates the start of a response.

[0074] In this embodiment, the second inference unit 130 may acquire, for each piece of target information, a token (e.g., one token) obtained in one inference process using the intermediate data cached in the state storage unit 102 and the machine learning model M. The second inference unit 130 may not execute the second or subsequent inference processes using the machine learning model M for each piece of target information.

[0075] The output information of the machine learning model M generated by the second inference unit 130 may include probability information. The probability information may be a probability distribution, a logit vector, a probability vector, or a probability value of 1 or more. The output information may include probability information for all tokens that the machine learning model M can process. Furthermore, the output information may be configured to include probability information for tokens requested to be generated by the candidate information (tokens corresponding to identification information of options, options, numbers, etc.), but not include probability information for other tokens. For example, if the candidate information is the above example 1, the output information may be configured to include probability information for three tokens, "1", "2", and "3", which are identification information of options, but not include probability information for other tokens.

[0076] The output information may also include tokens selected based on probability information. The selected tokens may be limited in advance to tokens requested to be generated by the candidate information (tokens corresponding to the identification information of the options, options, numbers, etc.). For example, among multiple tokens corresponding to the identification information of the options, the token for which the machine learning model M indicates the highest probability value or a token selected based on a probability distribution may be set as the output information corresponding to the target information. The output information corresponding to the target information may also be at least a part of the information output by the inference process by the machine learning model M, or may be information generated based on the output information.

[0077] The output unit 140 outputs an inference result for the inference request received by the request receiving unit 110 based on the output information generated by the second inference unit 130. The output unit 140 may output an inference result including output information for each of the multiple target information included in the inference request. The output unit 140 may transmit the inference result including the multiple output information to the terminal device 50. The output unit 140 may display the inference result including the multiple output information on the display device of the inference device 10. The inference result is an example of fifth information. Note that the inference result may include at least a portion of the output information, or may be information generated based on the output information. The inference result may be the same information as the output information.

[0078] The output unit 140 may include in the inference result, for each of the plurality of pieces of target information, the token showing the highest probability value or a token selected based on the probability distribution. The output unit 140 may determine information to be included in the inference result based on probability information included in the output information for each of the plurality of pieces of target information. The output unit 140 may include, in the inference result, tokens requested to be generated by the candidate information (tokens corresponding to identification information of options, options, numbers, etc.) and the probability values ​​of these tokens for each of the plurality of pieces of target information. The output unit 140 may include, in the inference result, a value calculated based on the probability information included in the output information as a probability value. As an example, the output unit 140 may normalize each probability value so that the sum of the probability values ​​of tokens corresponding to identification information of options is 1. Note that the second inference unit 130 may be configured to generate output information including the normalized probability values.

[0079] The output unit 140 may extract target information to be included in the inference result from the plurality of pieces of target information included in the inference request. The output unit 140 may extract target information to be included in the inference result based on the output information of each of the plurality of pieces of target information. As an example, the output unit 140 may extract target information to be included in the inference result by comparing a probability value included in the output information of each of the plurality of pieces of target information with a predetermined threshold. The output unit 140 may transmit the plurality of pieces of target information and the probability information to the terminal device 50 and cause the terminal device 50 to extract the target information.

[0080] In order for the output unit 140 to obtain only the probability information of tokens (tokens corresponding to the identification information of the options, the options, numbers, etc.) whose generation is requested by the candidate information, the machine learning model M may be configured to calculate only the probability information of tokens whose generation is requested by the candidate information in the final layer. As an example, the final layer of the machine learning model M according to this embodiment may be configured as a matrix of the number of tokens corresponding to the identification information of the options x the number of channels.

[0081] The final layer of a conventional large-scale language model is composed of a matrix of the number of all tokens x the number of channels. As an example, some conventional large-scale language models output approximately 50,000 tokens, resulting in a huge amount of calculation required to calculate output information and a huge amount of data for the output information. According to this embodiment, the number of tokens to be output by the machine learning model M can be limited based on the candidate information, thereby reducing the amount of calculation by the machine learning model M and the amount of output information transferred. Furthermore, according to this embodiment, probability information for all tokens is not output, making distillation of the machine learning model M difficult and protecting the machine learning model M. Furthermore, according to this embodiment, options, etc. can be determined using only the probability information of tokens (tokens corresponding to option identification information, options, numbers, etc.) whose generation is requested by the candidate information, thereby improving the accuracy of classification.

[0082] <prompt> A prompt, which is an example of model input information, will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of an inference request according to the first embodiment.

[0083] 3, an inference request 300 may include candidate information 310 and target information 320. The candidate information 310 may include question information 311 and option information 312. The question information 311 may include information indicating the content of the question ("Please judge your impression of the following sentence."). The option information 312 may include option identification information (1, 2, 3) and the content of the option (good impression, bad impression, neither).

[0084] The target information 320 may include multiple pieces of target information ("I went to this store but it was terrible" or "This book was interesting"). The target information 320 shown in Fig. 3 indicates the boundaries between pieces of target information by line break symbols or quotation marks, but the boundaries between pieces of target information may be indicated by any symbols including, for example, commas, colons, semicolons, etc. The quotation marks do not need to be included in the target information.

[0085] The first inference unit 120 may acquire the candidate information 310 from the inference request 300 and generate a prompt to be input to the machine learning model M. The second inference unit 130 may acquire any one of the target information 320 from the inference request 300 and generate a prompt to be input to the machine learning model M.

[0086] <Inference processing> The difference between the inference processing according to the comparative example and the inference processing according to the present embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram for explaining an example of the inference processing according to the comparative example. Fig. 5 is a diagram for explaining an example of the inference processing according to the first embodiment.

[0087] 4 and 5 schematically show forward processing by a Transformer included in machine learning model M. 4 and 5 illustrate some of the layers, processing, data, etc. of the Transformer included in machine learning model M. Machine learning model M may also include structures such as layers, processing, data, etc. that are not shown in FIG. 4 or 5.

[0088] As shown in FIG. 4, in the inference process according to the comparative example, multiple tokens x1, . . . , x n When input information including x1, ,x is input, n The first layer performs forward processing using each token x1,...,x n The query vector q corresponding to 1 1,···,q 1 n , key vector k 1 1,···,k 1 n , value vector v 1 1,···,v 1 n and use them to calculate the hidden vector h 1 1,···,h 1 n The hidden vector h output from the first layer 1 1,···,h 1 n is input to the second layer, which is the hidden vector h1 1,···,h 1 n The query vector q corresponding to 2 1,···,q 2 n , key vector k 2 1,···,k 2 n , value vector v 2 1,···,v 2 n and use them to calculate the hidden vector h 2 1,···,h 2 n The transformer repeatedly performs the same process in each layer, generating tokens x1, ,x n Next token x′2, ,x′ n+1 At this time, the transformer outputs the key vector k calculated in each layer. 1 1,···,k 2 1,... and the value vector v 1 1,···,v 2 1,··· is cached (dashed line).

[0089] Input information x1,...,x n When the inference process using n+1 The first layer performs inference processing using token x n+1 The query vector q corresponding to 1 n+1 , key vector k 1 n+1 , value vector v 1 n+1 , and the query vector q 1 n+1 , key vector k 1 1,···,k 1 n+1 and the value vector v 1 1,···,v 1 n+1 Using the hidden vector h 1 n+1 The hidden vector h output from the first layer 1 n+1is input to the second layer, which is the hidden vector h 1 n+1 The query vector q corresponding to 2 n+1 , key vector k 2 n+1 , value vector v 2 n+1 , and the query vector q 2 n+1 , key vector k 2 1,···,k 2 n+1 and the value vector v 2 1,···,v 2 n+1 Using the hidden vector h 2 n+1 The transformer repeats the same process in each layer and outputs the token x n+1 Next token x′ for n+2 At this time, the transformer outputs the key vector k calculated in each layer. 1 n+1 ,k 2 n+1 , and the value vector v 1 n+1 ,v 2 n+1 ,··· is cached (dashed line).

[0090] In this way, in the inference process according to the comparative example, in each layer of the Transformer, the next token for an input token is inferred using the query vector, key vector, and value vector corresponding to the latest token and the key vectors and value vectors corresponding to all cached past tokens. Therefore, the key vectors and value vectors calculated in each layer are cached for use in later forward processing. The key-value cache allows the machine learning model M to reuse previously calculated key vectors and value vectors, enabling high-speed execution of the inference process.

[0091] As shown in FIG. 5, in the inference process according to this embodiment, multiple tokens x1, . . . , xn When candidate information including x1,...,x is input, n The first layer performs forward processing using each token x1,...,x n The query vector q corresponding to 1 x1 ,···,q 1 xn , key vector k 1 x1 ,···,k 1 xn , value vector v 1 x1 ,···,v 1 xn and use them to calculate the hidden vector h 1 x1 ,···,h 1 xn The hidden vector h output from the first layer 1 x1 ,···,h 1 xn is input to the second layer, which is the hidden vector h 1 x1 ,···,h 1 xn The query vector q corresponding to 2 x1 ,···,q 2 xn , key vector k 2 x1 ,···,k 2 xn , value vector v 2 x1 ,···,v 2 xn and use them to calculate the hidden vector h 2 x1 ,···,h 2 xn The transformer repeatedly performs the same process in each layer, generating tokens x1, ,x n Next tokens x′2, x′3, , x′ n+1 At this time, the transformer outputs the key vector k calculated in each layer. 1 x1 ,···,k2 x1 , and the value vector v 1 x1 ,···,v 2 x1 ,··· is cached (dashed line).

[0092] Candidate information x1,...,x n When the inference process using the tokens y1, y m The target information 1 includes multiple tokens z1,...,z k Here, in the inference process according to this embodiment, the key vector k of each layer calculated in the inference process for the candidate information and cached in the state storage unit 102 is used. 1 x1 ,···,k 2 x1 , and the value vector v 1 x1 ,···,v 2 x1 , ... m The query vector q corresponding to 1 y1 ,···,q 1 ym , key vector k 1 y1 ,···,k 1 ym , value vector v 1 y1 ,···,v 1 ym , and the query vector q 1 y1 ,···,q 1 ym , key vector k 1 x1 ,···,k 1 xn ,k 1 y1 ,···,k 1 ym and the value vector v 1 x1,···,v 1 xn ,v 1 y1 ,···,v 1 ym Using the hidden vector h 1 y1 ,···,h 1 ym The hidden vector h output from the first layer 1 y1 ,···,h 1 ym is input to the second layer, which is the hidden vector h 1 y1 ,···,h 1 ym The query vector q corresponding to 2 y1 ,···,q 2 ym , key vector k 2 y1 ,···,k 2 ym , value vector v 2 y1 ,···,v 2 ym , and the query vector q 2 y1 ,···,q 2 ym , key vector k 2 x1 ,···,k 2 xn ,k 2 y1 ,···,k 2 ym and the value vector v 2 x1 ,···,v 2 xn ,v 2 y1 ,···,v 2 ym Using the hidden vector h 2 y1 ,···,h 2 ym The transformer repeatedly performs the same process in each layer to output the target information y1, ,y m Output information y′ form+1 Output.

[0093] At this time, the transformer uses the key vector k calculated in each layer. 1 y1 ,···,k 1 ym ,k 2 y1 ,···,k 2 ym and the value vector v 1 y1 ,···,v 1 ym ,v 2 y1 ,···,v 2 ym The transformer does not need to cache the target information y1,...,y m When the inference process using 1 y1 ,···,k 1 ym ,k 2 y1 ,···,k 2 ym and the value vector v 1 y1 ,···,v 1 ym ,v 2 y1 ,···,v 2 ym may be erased from memory. The transformer 1 y1 ,···,k 1 ym ,k 2 y1 ,···,k 2 ym and the value vector v 1 y1 ,···,v 1 ym ,v 2 y1 ,···,v 2 ym The transformer does not need to execute the instruction to cache the key vector k for subsequent inference processing. 1y1 ,···,k 1 ym ,k 2 y1 ,···,k 2 ym and the value vector v 1 y1 ,···,v 1 ym ,v 2 y1 ,···,v 2 ym It is also possible to execute an instruction not to cache the hidden vector h for subsequent inference processing. 1 y1 ,···,h 1 ym After the calculation of the output information y′ m+1 Before is calculated, the query vector q 1 y1 ,···,q 1 ym , key vector k 1 y1 ,···,k 1 ym and the value vector v 1 y1 ,···,v 1 ym can be erased from memory. Similarly, the hidden vector h 2 y1 ,···,h 2 ym After the calculation of the output information y′ m+1 Before is calculated, the query vector q 2 y1 ,···,q 2 ym , key vector k 2 y1 ,···,k 2 ym and the value vector v 2 y1 ,···,v 2 ym may be erased from memory.

[0094] The inference process using the target information 2 is performed in the same way as the inference process using the target information 1. That is, the first layer is composed of tokens z1,...,z k The query vector q corresponding to 1 z1 ,···,q 1 zk , key vector k 1 z1 ,···,k 1 zk , value vector v 1 z1 ,···,v 1 zk , and the query vector q 1 z1 ,···,q 1 zk , key vector k 1 x1 ,···,k 1 xn ,k 1 z1 ,···,k 1 zk , value vector v 1 x1 ,···,v 1 xn ,v 1 z1 ,···,v 1 zk Using the hidden vector h 1 z1 ,···,h 1 zk The hidden vector h output from the first layer 1 z1 ,···,h 1 zk is input to the second layer, which is the hidden vector h 1 z1 ,···,h 1 zk The query vector q corresponding to 2 z1 ,···,q 2 zk , key vector k 2 z1 ,···,k 2 zk , value vector v 2z1 ,···,v 2 zk , and the query vector q 2 z1 ,···,q 2 zk , key vector k 2 x1 ,···,k 2 xn ,k 2 z1 ,···,k 2 zk , value vector v 2 x1 ,···,v 2 xn ,v 2 z1 ,···,v 2 zk Using the hidden vector h 2 z1 ,···,h 2 zk The transformer repeatedly performs the same process in each layer to output the target information z1,...,z k Output information z′ for k+1 At this time, the transformer outputs the k 1 z1 ,···,k 1 zk ,k 2 z1 ,···,k 2 zk and the value vector v 1 z1 ,···,v 1 zk ,v 2 z1 ,···,v 2 zk There is no need to cache the hidden vector h 1 z1 ,···,h 1 zk After the calculation of the output information z′ k+1 Before is calculated, the query vector q 1 z1 ,···,q 1 zk , key vector k1 z1 ,···,k 1 zm and the value vector v 1 z1 ,···,v 1 zm can be erased from memory. Similarly, the hidden vector h 2 z1 ,···,h 2 zk After the calculation of the output information z′ k+1 Before is calculated, the query vector q 2 z1 ,···,q 2 zk , key vector k 2 z1 ,···,k 2 zk and the value vector v 2 z1 ,···,v 2 zk may be erased from memory.

[0095] In this embodiment, the key vectors and value vectors corresponding to candidate information are cached and shared in the inference process for each piece of target information, so that calculation of the key vectors and value vectors corresponding to candidate information can be omitted in the inference process for the target information. Furthermore, in the inference process for the target information, the query vectors, key vectors, and value vectors corresponding to each piece of target information do not need to be cached for subsequent inference processes, so the amount of memory usage required for the inference process for the target information can be reduced. Furthermore, reduced memory usage reduces the required memory bandwidth, making it easy to optimize even existing GPUs, etc.

[0096] <User Interface> The user interface provided by the inference device 10 will be described with reference to Figures 6 and 7. As an example, the user interface of the inference device 10 is displayed on a display device of the inference device 10 or the terminal device 50. In this embodiment, an example will be described in which each screen is displayed on a display device of the terminal device 50.

[0097] The user interface of the inference device 10 may include, for example, an input screen and an output screen. The input screen is a screen for inputting candidate information and target information. The output screen is a screen for displaying inference results for the target information.

[0098] <Input screen> Fig. 6 is a diagram showing an example of an input screen according to the first embodiment. As shown in Fig. 6, an input screen 400 may have a question input section 401, an option input section 402, an option addition button 403, an input information display section 404, and a target information input section 405.

[0099] The question input unit 401 receives input of a question to be included in the candidate information. The question input unit 401 may be automatically input by the information processing system 1000, or may be input or edited by the user of the information processing system 1000.

[0100] The option input unit 402 accepts input of options to be included in candidate information. The option input unit 402 may have an identification information input unit 411 and a description input unit 412. The identification information input unit 411 accepts input of identification information of the options. The identification information input unit 411 may accept selection of identification information from a predetermined list of identification information.

[0101] The option input unit 402 is an example of information for acquiring candidate information. The information for acquiring candidate information may include information for directly acquiring the candidate information, or may include information for acquiring information necessary for acquiring the candidate information.

[0102] The explanation input unit 412 accepts input of an explanation of the option (including the content of the option). The option input unit 402 may be automatically input by the information processing system 1000, or may be input or edited by a user of the information processing system 1000. As an example, when a user inputs information into the explanation input unit 412, identification information may be automatically input into the identification information input unit 411. Note that when identification information of an option is automatically assigned by the information processing system 1000, the identification information of the option may not be displayed on the input screen 400. The first inference unit 120 may use the assigned identification information of the option to generate candidate information including the identification information.

[0103] The identification information input unit 411 may be capable of inputting numbers (1 to 10, etc.), letters (A to Z, a to z, etc.), symbols, etc. When inputting automatically, the identification information input unit 411 may select identification information from information that can be represented using information (for example, one token) that can be generated in one inference process using the machine learning model M. When a user inputs identification information, the identification information input unit 411 may determine whether the input identification information is a token that can be generated in one inference process. As an example, the identification information input unit 411 may use an arbitrary tokenizer to determine whether the identification information input by the user can be generated in one inference process.

[0104] The identification information input unit 411 may determine whether the identification information can be generated in one inference process and notify the user of the determination result. As an example, if the identification information cannot be expressed by a token (e.g., one token) that can be generated in one inference process, the input screen 400 may notify the user that the identification information cannot be set. The notification that the identification information cannot be set may be, for example, a pop-up error message or a beep sound.

[0105] The option input unit 402 may accept from the user an option that can be represented using information (for example, one token) that can be generated in one inference process using the machine learning model M. In this case, the option input unit 402 does not need to acquire identification information via the identification information input unit 411. Furthermore, if the accepted option cannot be represented by information that can be generated in one inference process, the user may be notified of this.

[0106] The add option button 403 is a button for adding a new option to the option input section 402. When the user presses the add option button 403, an identification information input section 411 and a description input section 412 are added to the option input section 402. Identification information that does not overlap with existing identification information input sections 411 may be automatically input into the added identification information input section 411. Note that the input screen 400 may have a delete button for deleting options included in the option input section 402.

[0107] The input screen 400 includes a question input section 401, an option input section 402, and an option addition button 403, so that the questions and options can be changed on demand. This allows the user to use any task at any time via the input screen 400.

[0108] 6 includes two options, but may include three or more options. Also, option input section 402 shows options with contradictory content (food-related, unrelated to food), but the options do not have to be contradictory.

[0109] The input information display unit 404 displays the model input information. The input information display unit 404 may automatically generate the model input information based on the information input to the question input unit 401 and the option input unit 402. The input screen 400 may have a button for generating the model input information to be displayed in the input information display unit 404. The input information display unit 404 may display the model input information in a format that can be edited by the user using a text editor or the like.

[0110] The input information display section 404 may include reference answer information 413. The reference answer information 413 may include a combination of target information and example answers. The reference answer information 413 may include a plurality of sets of target information and example answers. The reference answer information 413 may be generated based on information from the user, or may be generated based on information input to the question input section 401 and the option input section 402. The reference answer information 413 may be generated based on the machine learning model M or another machine learning model.

[0111] The input information display section 404 may include a placeholder 414. The placeholder 414 may include a placeholder ({{query}}) for embedding target information. The placeholder 414 may include information instructing the start of answering (for example, "answer"). The information instructing the start of answering may be a predetermined fixed sentence. The information instructing the start of answering may be omitted. As an example, the placeholder 414 shown in FIG. 6 may be only a placeholder ({{query}}).

[0112] The target information input unit 405 accepts input of target information. The target information input unit 405 may accept input of multiple pieces of target information. The target information input unit 405 may be input or edited by a user. Note that although the target information input unit 405 in FIG. 6 shows target information masked with X, Y, and Z, text written in a natural language sentence or the like may also be input.

[0113] The target information input unit 405 may accept input of an electronic file in which the target information is described. As an example, the target information input unit 405 may accept input of the target information by a user's drag-and-drop operation of the electronic file.

[0114] The electronic file may be recorded in a predetermined file format, such as a text file, an image file, a video file, an audio file, a spreadsheet file, a presentation file, a PDF (Portable Document Format) file, etc.

[0115] The target information may be written in a predetermined format in the electronic file. Examples of predetermined formats may include CSV (Comma Separated Value) format, Markdown notation, HTML (Hyper Text Markup Language), XML (Extensible Markup Language), JSON (JavaScript Object Notation) format, and JSONL (JSON Lines) format. When the target information is written in CSV format, the column name (e.g., "Text") may be written on the first line of the electronic file, and the target information may be written on the second and subsequent lines. When the target information is written in JSON or JSONL format, the target information may be written as an array.

[0116] The electronic file may include multiple columns. For example, when performing a task of evaluating a translation result, the electronic file may include a column indicating a sentence written in a first language and a column indicating a sentence written in a second language. Each piece of target information for the task of evaluating a translation result may be written as follows. In the example below, {en} is a placeholder for embedding a sentence written in English, and {ja} is a placeholder for embedding a sentence written in Japanese. Example 1: English: {en} English: {ja} Grade:"

[0117] Information retrieved from an external data source may be input to the target information input unit 405. Examples of the external data source may include a social networking service, a search engine, a database, a website, etc. The external data source may include multiple data sources.

[0118] The target information input unit 405 is an example of information for acquiring target information. The information for acquiring target information may include information for directly acquiring the target information, or may include information for acquiring information necessary for the target information.

[0119] When model input information is displayed in input information display section 404 and target information is input into target information input section 405, input screen 400 may generate model input information by embedding the target information input into target information input section 405 into placeholder 414, and transmit the generated model input information as an inference request to inference device 10. Input screen 400 may have a button for transmitting an inference request.

[0120] 6, the portion of the input information display unit 404 excluding the placeholder 414 corresponds to the candidate information included in the inference request. Therefore, the first inference unit 120 of the inference device 10 may calculate intermediate data (key-value cache) by inputting the portion of the input information display unit 404 excluding the placeholder 414 to the machine learning model M. Furthermore, the second inference unit 130 of the inference device 10 may generate output information for each of the multiple pieces of target information by batch processing the multiple pieces of target information input to the target information input unit 405. In this case, the second inference unit 130 may input the key-value cache generated by the first inference unit 120 to the machine learning model M.

[0121] <<Output screen>> 7A and 7B are diagrams showing examples of output screens. Fig. 7A shows output screen 500A, which displays the probability value for each option for each piece of target information. Fig. 7B shows output screen 500B, which displays the option with the highest probability value for each piece of target information. Fig. 7C shows output screen 500C, which displays the option with the highest probability value and the probability value of that option for each piece of target information. Fig. 7D shows output screen 500D, which displays the option with the highest probability value and the probability value of that option for each piece of target information, sorted in descending order of probability value.

[0122] The probability values ​​displayed on the output screen 500 may be only the probability values ​​of the tokens corresponding to the options. In this case, the probability values ​​of the tokens other than the options may be discarded. In FIG. 7, the probability values ​​are displayed as numbers between 0 and 1, but as another example, they may be displayed as percentages or in other formats that allow the high-low relationship between the options to be understood. In FIG. 7, the inference results are displayed for all the target information, but it is also possible to display only the inference results for some of the target information. As an example, only a predetermined number of target information with high probability values ​​may be displayed.

[0123] In FIG. 7(A), the sum of the probability values ​​for each option is normalized to 1, but the sum may be normalized to a different value, or may not be normalized at all. In FIG. 7(A), the probability values ​​for all options are displayed, but it is also possible to display only the probability values ​​for some options. The options to be displayed may be selected based on the probability values. In FIG. 7(D), the options are sorted in descending order of probability value, but they may also be sorted in ascending order of probability value.

[0124] Although probability values ​​are displayed for all options in FIG. 7A, probability values ​​may be displayed only for some options. For example, in FIG. 7A, only the probability value for option A may be displayed. In this case, the target information may be sorted in descending or ascending order based on the probability value of option A. Also, only the target information for which the probability value of option A is equal to or greater than a predetermined threshold may be displayed. In this case, the target information for which the probability value of option A is equal to or greater than a predetermined threshold may be sorted in descending or ascending order based on the probability value of option A. Also, information based on an option selected based on the probability information may be displayed as an inference result on the output screen 500. For example, a predetermined symbol or the like corresponding to the selected option may be displayed on the output screen 500 as the inference result. Also, when output information related to multiple categories is generated for one piece of target information, the inference results for each of the multiple categories may be displayed simultaneously on the output screen 500.

[0125] The output screen 500 may save the inference results in a predetermined file format, which may be the same as the electronic file used to input the target information into the input screen 400.

[0126] <Other screens> The user interface of inference device 10 may include a screen for selecting a machine learning model to be used to perform a predetermined task from a plurality of machine learning models. The user interface of inference device 10 may include a screen for inputting setting information related to the selectable machine learning models.

[0127] <Flow of inference processing> The inference processing executed by the information processing system 1000 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the inference processing according to the first embodiment.

[0128] In step S1, the terminal device 50 transmits an inference request to the inference device 10 in response to a user's operation on the input screen 400. The inference request includes the candidate information and multiple pieces of target information input on the input screen 400.

[0129] The request receiving unit 110 of the inference device 10 receives an inference request from the terminal device 50. The request receiving unit 110 accepts the received inference request. The request receiving unit 110 sends candidate information included in the inference request to the first inference unit 120. The request receiving unit 110 sends multiple pieces of target information included in the inference request to the second inference unit 130.

[0130] In step S2, the first inference unit 120 of the inference device 10 receives candidate information from the request receiving unit 110. The first inference unit 120 generates model input information based on the candidate information. The first inference unit 120 reads out the machine learning model M from the model storage unit 101. The first inference unit 120 inputs the model input information based on the candidate information to the machine learning model M.

[0131] In step S3, the first inference unit 120 inputs the model input information to the machine learning model M and executes the inference process. Here, the first inference unit 120 calculates intermediate data using the candidate information. The first inference unit 120 stores the calculated intermediate data in the state storage unit 102 (caches the intermediate data).

[0132] In step S4, the second inference unit 130 of the inference device 10 receives multiple pieces of target information from the request receiving unit 110. The second inference unit 130 acquires one piece of target information from the multiple pieces of target information. The second inference unit 130 generates model input information based on the acquired target information.

[0133] The second inference unit 130 reads intermediate data calculated from the candidate information from the state storage unit 102. The second inference unit 130 reads the machine learning model M from the model storage unit 101. The second inference unit 130 inputs model input information based on the target information to the machine learning model M.

[0134] In step S5, the second inference unit 130 executes a single inference process using the machine learning model M to generate output information for the target information. Here, the second inference unit 130 executes the inference process using the intermediate data read from the state storage unit 102 in step S4. The second inference unit 130 acquires output information from the machine learning model M. The second inference unit 130 sends the acquired output information to the output unit 140. The second inference unit 130 may not execute an inference process subsequent to the single inference process for the target information. The second inference unit 130 may not store the intermediate data calculated in the inference process for the target information (the intermediate data may not be cached). As a non-limiting example, the second inference unit 130 may be configured to execute only one inference process for each piece of target information.

[0135] The second inference unit 130 executes steps S4 and S5 for each of the multiple pieces of target information included in the inference request. The second inference unit 130 may batch process the multiple pieces of target information in steps S4 and S5. The second inference unit 130 may repeatedly execute batch processing of a predetermined number of pieces of target information until output information is generated for all of the target information.

[0136] In step S6, the output unit 140 of the inference device 10 receives output information for the target information from the second inference unit 130. Upon receiving the output information for each of the multiple pieces of target information, the output unit 140 generates an inference result based on the output information. The output unit 140 transmits the generated inference result to the terminal device 50.

[0137] The terminal device 50 receives the inference result from the inference device 10. The terminal device 50 generates an output screen 500 based on the received inference result. The terminal device 50 displays the output screen 500 on a display device.

[0138] [Second embodiment] The output information generated by the inference device 10 according to the first embodiment can be used as learning data for generating another model. In the second embodiment, a configuration is described in which the output information generated by the inference device 10 according to the first embodiment is used to generate another model. The other model may be a machine learning model such as linear regression or a neural network, or may be another mathematical model.

[0139] In this embodiment, the request receiving unit 110 receives an inference request including multiple pieces of candidate information and multiple pieces of target information. For example, first candidate information among the multiple pieces of candidate information may include options related to a first evaluation index. For example, second candidate information different from the first candidate information may include options related to a second evaluation index different from the first evaluation index.

[0140] In this embodiment, the first inference unit 120 calculates intermediate data for each of the multiple candidate information included in the inference request. The second inference unit 130 executes an inference process using the intermediate data for the first candidate information and an inference process using the intermediate data for the second candidate information for each of the multiple target information included in the inference request. The output unit 140 outputs output information including a probability value of an option for the first evaluation index and a probability value of an option for the second evaluation index for each of the multiple target information.

[0141] 9 is a diagram showing a first example of a prompt according to the second embodiment. As shown in FIG. 9, prompt 310A includes a question asking whether the target information is violent ("Does the following sentence contain violent language?") and answer options for the question (very violent, violent, not violent). In other words, prompt 310A is model input information that includes options for an index for evaluating the violence of the target information.

[0142] 10 is a diagram showing a second example of a prompt according to the second embodiment. As shown in FIG. 10, prompt 310B includes a question asking whether the target information is false ("Does the following sentence contain false content?") and answer options for the question (yes, no). In other words, prompt 310B is model input information that includes options for an index for evaluating the veracity of the target information.

[0143] The inference device 10 may generate another model based on the output information. As an example, the other model may be a model that receives a probability value of an option related to a first evaluation index and a probability value of an option related to a second evaluation index as input, and outputs a probability value of an option related to a third evaluation index. The probability value of an option related to the third evaluation index is an example of second output information.

[0144] By way of example, information input to other models may include the following: Very violent 0.40 / violent 0.15 / non-violent 0.45 (first evaluation index) Yes 0.5 / No 0.5 (second evaluation index)

[0145] The information output from other models is as follows: Good impression 0.05 / bad impression 0.95

[0146] By adding correct answer information of the third evaluation index to the output information output by the inference device 10, it is possible to generate training data for other models. The correct answer information may be obtained from existing evaluation results. If there are no existing evaluation results, the correct answer information may be added manually. The correct answer information may be calculated according to predetermined rules or may be predicted using another machine learning model.

[0147] As an example, the correct answer information may be calculated by the following formula: Probability of "Good Impression" = sigmoid(-2.0 * Probability of "Very Violent" - Probability of "Violent" + 3.0 * Probability of "Not Violent" - Probability of "Yes" + Probability of "No") Probability of a "bad impression" = 1 - Probability of a "good impression"

[0148] The other model may be, for example, a linear regression model. In this embodiment, the inference device 10 outputs a probability value of the option for each evaluation index, so that the linear regression model can be easily generated. However, the other model is not limited to the linear regression model, and any machine learning model may be generated.

[0149] When evaluating indicators that can be evaluated along various evaluation axes, the evaluator can evaluate them along any evaluation axis based on their own subjective judgment. Therefore, if the evaluator subjectively assigns correct answer information, it may not be possible to generate a model that can stably evaluate the indicator. According to the inference process of this embodiment, multiple intermediate evaluation results corresponding to multiple evaluation axes can be obtained, making it possible to generate a model that can stably evaluate even indicators that can be evaluated along various evaluation axes.

[0150] In this embodiment, the output information regarding the first evaluation index and the output information regarding the second evaluation index are generated by separate inference processes, but the output information regarding multiple evaluation indexes may be generated by a single inference process. A configuration for generating output information regarding multiple evaluation indexes by a single inference process will be described in the fourth embodiment.

[0151] [Third embodiment] The inference process according to the first embodiment can be applied to a technology called Retrieval Augmented Generation (RAG), which includes reference information obtained by searching a specified data source in a prompt in order to obtain good output results from a large-scale language model.

[0152] In the search expansion generation, reference information is searched for from a predetermined data source based on instruction information that instructs the execution of a task, and a prompt including the instruction information and the search results of the reference information is input to a large-scale language model. The large-scale language model performs the task while taking into account reference information that is not included in the training data, so it can generate data appropriate for the instruction information.

[0153] In search expansion generation, the more appropriate the reference information included in the prompt is for the instruction information, the better the data that can be generated. However, the search accuracy of data sources varies greatly depending on, for example, the query, search conditions, type of data source, search method, etc.

[0154] In this embodiment, in the framework of search expansion generation, the inference process according to the first embodiment is used to evaluate whether reference information is appropriate. By selecting reference information evaluated as appropriate from the search results and including it in the prompt, appropriate data can be generated for the instruction information entered by the user.

[0155] <Overall configuration of information processing system> The overall configuration of the information processing system according to this embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the overall configuration of the information processing system according to the third embodiment.

[0156] 11, the information processing system 1000 includes a generating device 20, a searching device 30, and a terminal device 50. The generating device 20, the searching device 30, and the terminal device 50 may be connected to each other via a communication network such as a LAN (Local Area Network) or the Internet so as to be able to communicate data with each other.

[0157] The generating device 20 is an example of an information processing device such as a personal computer, a workstation, or a server that executes a generation task to generate predetermined data in response to a generation request from the terminal device 50. The generating device 20 may receive a generation request from the terminal device 50. The generating device 20 may transmit a generation result in response to the generation request to the terminal device 50.

[0158] A generation request is information or a signal that requests the generation of specified data. The generation request may include information input by a user and search criteria set by the user. Hereinafter, information input by a user will also be referred to as "user-input information."

[0159] The generation device 20 may acquire search results for reference information from the retrieval device 30 based on the generation request received from the terminal device 50. The generation device 20 may transmit a search request to the retrieval device 30 based on the generation request.

[0160] A search request is information or a signal requesting a search for reference information. The search request may include, for example, a query and search conditions. The query may include user-input information included in the generation request. The query may be generated based on the user-input information. The search conditions may include search conditions included in the generation request. The search conditions may be determined based on the user-input information.

[0161] The generation device 20 may generate model input information based on the search results of the reference information acquired from the search device 30. The generation device 20 may generate output information for each piece of reference information included in the search results by inputting the model input information to the machine learning model M. The generation device 20 may select one or more pieces of reference information to be used for generating output information in response to a generation request, based on the output information for each piece of reference information.

[0162] The generation device 20 may generate model input information based on the user input information and one or more selected reference information. The generation device 20 may generate output information for the generation request by inputting the model input information to the machine learning model M. The generation device 20 may generate output information for the generation request by inputting the model input information to another machine learning model. The output information for the generation request is an example of third output information. The generation device 20 may transmit a generation result for the generation request to the terminal device 50 based on the output information for the generation request.

[0163] The search device 30 is an example of an information processing device such as a personal computer, workstation, or server that searches for reference information in response to a search request from the generation device 20. The search device 30 may include a data source D that stores various pieces of reference information. As an example, the data source D may be a storage device or a database in which the reference information is stored. The data source D may also be included in an external storage device, information processing device, or information processing system.

[0164] The search device 30 may be implemented in multiple devices or systems with different data sources D. The search device 30 may be implemented in one device or system with multiple data sources D. The search device 30 may also be an external device or system. The search device 30 may search for reference information from external data sources D.

[0165] The data source D may be built into the generation device 20. The data source D may be distributed and held in an external information processing system consisting of multiple devices. In this case, the information processing system 1000 may not include the search device 30.

[0166] The information processing system 1000 may include a plurality of devices, such as one or more of the generating device 20, the searching device 30, and the terminal device 50. The generating device 20 or the searching device 30 may be realized by a plurality of computers, or may be realized as a cloud computing service. The classification of the devices, such as the generating device 20, the searching device 30, and the terminal device 50 shown in FIG. 11, is an example.

[0167] <Functional configuration of the generation device> The functional configuration of the generation device 20 will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the functional configuration of the generation device according to the third embodiment.

[0168] 12, the generation device 20 includes a model storage unit 101, a state storage unit 102, a request receiving unit 110, a first inference unit 120, a second inference unit 130, an output unit 140, a search unit 210, a selection unit 220, and a generation unit 230. The generation device 20 differs from the inference device 10 according to the first embodiment in that it further includes the search unit 210, the selection unit 220, and the generation unit 230.

[0169] The following describes the generating device 20 according to this embodiment, focusing on the differences from the inference device 10 according to the first embodiment.

[0170] In this embodiment, the request receiving unit 110 receives a generation request. The generation request may include user-input information and search conditions. The user-input information and search conditions may be input by the user to the terminal device 50, or may be automatically generated by the generation device 20 or the terminal device 50.

[0171] The user-input information may include search terms to include in the query. The user-input information may include information that can identify the information desired by the user. The user-input information may include one or more natural language sentences that describe the information desired by the user. The user-input information may be, for example, instructions, questions, etc., such as "Please give me an overview of Company A" or "Please create a summary of recent news about generative AI (artificial intelligence)."

[0172] The user input information may include text data, structured data, image data, or audio data. Image data may be, for example, a still image or a video. Text data may be text data generated by speech recognition of audio data or video. Text data may be text generated by character recognition of image data. Audio data may be audio data generated by speech synthesis of text data. Structured data may be, for example, a table or graph.

[0173] As an example, the search conditions may include at least one of information indicating the data source to be searched, the type of data source, the time range of the creation date, attribute information of the reference information to be searched, or the number of items to be included in the search results.

[0174] The information indicating the data source may be information that can identify the data source. For example, the information that can identify the data source may be identification information that identifies the data source, or information that indicates the location of the data source (e.g., a host name, an Internet Protocol (IP) address, a connection string, a Uniform Resource Locator (URL), etc.).

[0175] The search unit 210 searches for reference information based on the generation request received by the request receiving unit 110. For example, the search unit 210 may transmit a search request including user input information included in the generation request to the search device 30. The search unit 210 may receive search results transmitted by the search device 30 in response to the search request. The search results may include multiple pieces of reference information that match the search conditions indicated in the search request.

[0176] The search request may include at least one of a query and a search condition. The query may be generated based on user input information included in the generation request. As an example, the query may include a keyword included in the user input information. The search condition may include a search condition included in the generation request. The search condition may include a predetermined search condition. The search condition may be determined based on user input information included in the generation request.

[0177] The search unit 210 may search for reference information from multiple data sources specified in the search conditions. The search unit 210 may include the number of pieces of reference information specified in the search conditions in the search results. The search unit 210 may include a predetermined number of pieces of reference information in the search results. The search unit 210 may set the number of pieces of reference information to be included in the search results depending on the amount of data that can be input to the machine learning model M. The number of pieces of reference information may be 10,000, for example.

[0178] In this embodiment, the first inference unit 120 generates candidate information based on a generation request received by the request receiving unit 110. The first inference unit 120 may generate candidate information based on user input information included in the generation request.

[0179] For example, the candidate information may include a question asking whether the reference information acquired by the search unit 210 is appropriate for the user-input information. For example, the candidate information may include options regarding an index for evaluating the appropriateness of the reference information.

[0180] In this embodiment, the second inference unit 130 generates output information for each of the multiple pieces of reference information included in the search results acquired by the search unit 210, based on the intermediate data calculated by the first inference unit 120 and the machine learning model M stored in the model storage unit 101. The second inference unit 130 may read the intermediate data stored in the state storage unit 102 and generate output information for one piece of reference information included in the search results by inputting the intermediate data and one piece of reference information included in the search results to the machine learning model M. The second inference unit 130 may generate output information in parallel for two or more pieces of reference information included in the multiple pieces of reference information.

[0181] The second inference unit 130 may divide one piece of reference information into multiple parts and input them to the machine learning model M. The second inference unit 130 may generate output information for the multiple divided parts in parallel. As an example, the second inference unit 130 may divide the reference information in units of one sentence, a predetermined number of sentences, a predetermined number of characters, one or more paragraphs, or one or more chapters. When dividing the reference information, the second inference unit 130 may divide the reference information so that some of the pieces overlap. When the reference information is multimodal data, the second inference unit 130 may divide the reference information in units of text data, image data, audio data, etc.

[0182] The selection unit 220 selects one or more pieces of reference information to be used for generating output information in response to the generation request from the search results for reference information acquired by the search unit 210, based on the output information generated by the second inference unit 130. The selection unit 220 may acquire information that can identify the reference information. The information that can identify the reference information may be, for example, identification information that identifies the reference information, or information that indicates the location of a file that describes the reference information.

[0183] The selection unit 220 may select reference information whose probability value of a predetermined option is equal to or greater than a predetermined threshold. The selection unit 220 may select a predetermined number of pieces of reference information from the top of the probability value of the predetermined option. The predetermined number may be 10, for example. The predetermined options may be options that indicate that the reference information is appropriate, for example.

[0184] The generation unit 230 generates output information for the generation request based on the generation request received by the request receiving unit 110 and one or more pieces of reference information selected by the selection unit 220. As an example, the generation unit 230 may generate model input information based on the generation request and the reference information.

[0185] The generation unit 230 may generate the model input information using at least a portion of information included in the one or more pieces of reference information selected by the selection unit 220. The generation unit 230 may generate the model input information using information obtained by processing the reference information. The generation unit 230 may generate the model input information using at least a portion of information included in the user input information. The generation unit 230 may generate the model input information using information obtained by processing the user input information.

[0186] The generation unit 230 may generate the model input information by embedding the user input information and the selected one or more pieces of reference information into a predetermined template. The generation unit 230 may generate the model input information by processing the template, the user input information, and the reference information based on predetermined rules. The generation unit 230 may generate the model input information by inputting the template, the user input information, and the reference information into the machine learning model M or another machine learning model.

[0187] A template for generating model input information may include one or more placeholders for embedding user input information or one or more pieces of reference information selected by the selection unit 220. The template may include instruction information for instructing the generation of output information for the generation request. The template may also include constraint information regarding the output information for the generation request. The instruction information and constraint information may be predetermined fixed sentences.

[0188] The generation unit 230 may generate output information for the generation request by inputting model input information to the machine learning model M. The generation unit 230 may generate output information for the generation request by inputting model input information to another machine learning model different from the machine learning model M. The generation unit 230 may generate output information for the generation request by inputting model input information to one machine learning model selected from multiple machine learning models. The machine learning model used to generate output information for the generation request may, for example, be specified by the generation request. The machine learning model specified in the generation request may be selected by the user.

[0189] The output unit 140 outputs a generation result for the generation request accepted by the request accepting unit 110, based on the output information for the generation request generated by the generation unit 230. The output unit 140 may transmit the generation result including the output information for the generation request to the terminal device 50. The output unit 140 may display the generation result including the output information for the generation request on the display device of the generation device 20.

[0190] The generation results may include text data, image data, or audio data. The text data may be, for example, natural language text. The image data may be, for example, a still image or a video. The text data may be text data obtained by speech recognition of audio data or audio data recorded in a video. The text data may be text obtained by character recognition of image data. The audio data may be audio data obtained by speech synthesis of text data.

[0191] <prompt> A prompt, which is an example of model input information according to the third embodiment, will be described with reference to FIGS. 13 and 14. FIG.

[0192] 13 is a diagram showing a first example of a prompt according to the third embodiment. A prompt 310C shown in FIG. 13 is an example of model input information that the first inference unit 120 inputs to the machine learning model M.

[0193] As shown in FIG. 13, prompt 310C includes a question asking whether the user-input information (Question A) is appropriate ("Please judge whether the following reference information is appropriate as information to refer to when answering the question {Question A}"). Prompt 310C includes options (appropriate, inappropriate, etc.) for evaluating the appropriateness of the reference information. Note that {Question A} is a placeholder into which the user-input information is embedded.

[0194] Fig. 14 is a diagram showing a second example of a prompt according to the third embodiment. A prompt 330 shown in Fig. 14 is an example of model input information that the generation unit 230 inputs to the machine learning model M or another machine learning model.

[0195] As shown in FIG. 14, prompt 330 includes a generation instruction ("Please use the following reference information to answer the question {Question A}") using user-input information (Question A). Prompt 330 includes reference information (Reference Information A, Reference Information B, Reference Information C) selected by selection unit 220. Note that {Question A} is a placeholder into which the user-input information is embedded. {Reference Information A}, {Reference Information B}, and {Reference Information C} are placeholders into which the reference information is embedded.

[0196] <Generation process flow> The generation process executed by the information processing system 1000 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the generation process according to the third embodiment.

[0197] In step S11, the terminal device 50 transmits a generation request to the generation device 20 in response to an operation by the user. The generation request includes information input by the user. The generation request may also include search conditions.

[0198] The request receiving unit 110 of the generation device 20 receives a generation request from the terminal device 50. The request receiving unit 110 accepts the received generation request. The request receiving unit 110 sends the user input information and search conditions to the search unit 210. The request receiving unit 110 sends the user input information to the first inference unit 120 and the generation unit 230.

[0199] In step S12, the search unit 210 of the generation device 20 receives the user-input information and search conditions from the request receiving unit 110. The search unit 210 transmits a search request to the search device 30 based on the user-input information and search conditions. The search request includes a query and search conditions set based on the user-input information.

[0200] The search device 30 receives a search request from the generation device 20. Based on the search request, the search device 30 searches the data source D for reference information that satisfies the search conditions. The search device 30 transmits the search results for the reference information to the generation device 20. Note that the search device 30 may search for the reference information using any method based on the search request.

[0201] The search unit 210 of the generation device 20 receives the search results for the reference information from the search device 30. The search unit 210 sends the search results for the reference information to the second inference unit 130 and the selection unit 220.

[0202] In step S13, the first inference unit 120 of the generation device 20 receives user input information from the request receiving unit 110. The first inference unit 120 generates candidate information based on the user input information. The first inference unit 120 generates model input information based on the candidate information. The first inference unit 120 reads out the machine learning model M from the model storage unit 101. The first inference unit 120 inputs the model input information based on the candidate information to the machine learning model M.

[0203] In step S14, the machine learning model M executes an inference process using the input model input information. The machine learning model M calculates intermediate data using the candidate information. The first inference unit 120 acquires the intermediate data generated by the machine learning model M. The first inference unit 120 stores (caches) the acquired intermediate data in the state storage unit 102.

[0204] In step S15, the second inference unit 130 of the generation device 20 receives the search results for reference information from the search unit 210. The second inference unit 130 acquires one piece of reference information from the multiple pieces of reference information included in the search results. The second inference unit 130 generates model input information based on the acquired reference information.

[0205] The second inference unit 130 reads cached intermediate data from the state storage unit 102. The second inference unit 130 reads the machine learning model M from the model storage unit 101. The second inference unit 130 inputs model input information based on the intermediate data and reference information corresponding to the candidate information to the machine learning model M. Note that the second inference unit 130 may input the model input information to a machine learning model other than the machine learning model M to execute the inference process.

[0206] In step S16, the machine learning model M executes an inference process using the input model input information. The machine learning model M generates output information for the reference information. The second inference unit 130 acquires the output information generated by the machine learning model M. The second inference unit 130 sends the acquired output information to the selection unit 220. The second inference unit 130 may generate the output information by performing a single inference process on the reference information.

[0207] The second inference unit 130 executes steps S15 and S16 for each of the multiple pieces of reference information included in the search results for reference information. The second inference unit 130 may batch-process the multiple pieces of reference information in steps S15 and S16. The second inference unit 130 may repeatedly execute batch processing of a predetermined number of pieces of reference information until output information is generated for all of the reference information.

[0208] In step S17, the selection unit 220 of the generation device 20 receives the search results for the reference information from the search unit 210. The selection unit 220 receives output information for the reference information from the second inference unit 130. Upon receiving the output information for each of the multiple pieces of reference information, the selection unit 220 selects reference information to be used for generating output information for the generation request from the search results for the reference information based on the output information. The selection unit 220 sends the selected reference information to the generation unit 230. Here, the selection unit 220 may select the reference information to be used for generating output information for the generation request based on a probability value for a predetermined option included in the output information.

[0209] In step S18, the generation unit 230 of the generation device 20 receives the user input information from the request receiving unit 110. The generation unit 230 receives the selected reference information from the selection unit 220. The generation unit 230 generates model input information based on the user input information and the selected reference information.

[0210] In step S19, the generation unit 230 of the generation device 20 reads out the machine learning model M from the model storage unit 101. The generation unit 230 inputs the model input information generated in step S18 into the machine learning model M.

[0211] The machine learning model M executes a predetermined task (such as a generation process) based on the input model input information, and generates output information for the generation request. The generation unit 230 acquires the output information for the generation request from the machine learning model M. The generation unit 230 sends the acquired output information for the generation request to the output unit 140.

[0212] In step S20, the output unit 140 of the generation device 20 receives output information in response to the generation request from the generation unit 230. The output unit 140 generates a generation result based on the output information in response to the generation request. The output unit 140 transmits the generated generation result to the terminal device 50.

[0213] The terminal device 50 receives the generation result from the generation device 20. The terminal device 50 displays the received generation result on a display device. The generation result may include a part of the output information generated by the generation unit 230, or may include information generated based on the output information.

[0214] [Fourth embodiment] In the first embodiment, as an example, a configuration has been described in which one token that can be generated in one inference process is generated as output information. In the fourth embodiment, a configuration will be described in which output information regarding multiple tokens is generated in one inference process. According to this embodiment, for example, output information regarding multiple evaluation indexes shown in the second embodiment can be generated in one inference process.

[0215] In this embodiment, as an example, a classification task is executed to classify each of a plurality of pieces of target information into options related to a first classification and options related to a second classification. In this embodiment, the options related to the first classification include options A, B, and C, and the options related to the second classification include options D and E. Therefore, the output information generated by the inference device 10 may include probability values ​​AD, AE, BD, BE, CD, and CE for each of the plurality of pieces of target information. Here, AD means selecting option A for the first classification and option D for the second classification, respectively.

[0216] In this embodiment, the candidate information may include information that explicitly instructs the answering method. As an example, the candidate information may include information that instructs consecutive output of one of options A, B, and C related to the first classification and one of options D and E related to the second classification. The information that instructs the answering method may be a predetermined fixed sentence.

[0217] In this embodiment, the classification task may be achieved by inputting an attention mask to the machine learning model M. The attention mask is information that masks references to some tokens.

[0218] FIG. 16 is a diagram illustrating an example of an attention mask. As shown in FIG. 16, candidate information, target information, and options A, B, and C related to the first classification are input to the machine learning model M. Note that the candidate information may be intermediate data (key-value cache) calculated using the candidate information. The attention mask is set so that A, B, and C each refer only to the candidate information and target information.

[0219] Specifically, the attention mask is set as follows: ·A refers to all past tokens (candidate information, target information). ·B refers to past tokens (candidate information, target information) excluding option A. ·C refers to past tokens (candidate information, target information) excluding options A and B.

[0220] The next token for A is inferred by forward processing using "candidate information, target information, A." The next token for B is inferred by forward processing using "candidate information, target information, B." The next token for C is inferred by forward processing using "candidate information, target information, C." As a result, the probability information for each of AD, AE, BD, BE, CD, and CE can be generated in a single inference process.

[0221] Note that the positional embedding may be changed when setting the attention mask shown in Fig. 16. However, sufficient approximate results can be obtained without changing the positional embedding.

[0222] Here, a configuration has been described in which output information regarding two tokens is generated in one inference process, but it may be configured to generate output information regarding three or more tokens. As an example, a configuration in which output information regarding three tokens is generated will be described.

[0223] The options for the first classification include options A, B, and C, the options for the second classification include options D and E, and the options for the third classification include options F and G. Therefore, the inference results output by the inference device 10 may include probability information for each of the multiple target information items ADF, ADG, AEF, AEG, BDF, BDG, BEF, BEG, CDF, CDG, CEF, and CEG.

[0224] The machine learning model M receives the target information with the token string ADAEBDBECDCE added to the end. The attention mask is set as follows: The first token A refers to all past tokens (candidate information, target information). The second token, D, refers to all previous tokens (candidate information, target information, A). The third token, A, refers to past tokens (candidate information, target information) excluding AD. The fourth token E refers to past tokens (candidate information, target information, A) excluding AD. The fifth token, B, refers to past tokens (candidate information, target information) excluding ADAE. · The sixth token, D, refers to past tokens (candidate information, target information, B) excluding ADAE. The seventh token B refers to past tokens (candidate information, target information) excluding ADAEBD. · The eighth token E refers to past tokens (candidate information, target information, B) except ADAEBD. The ninth token, C, refers to past tokens (candidate information, target information) excluding ADAEBDBE. · The tenth token D refers to past tokens (candidate information, target information, C) except ADAEBDBE. · The 11th token C refers to past tokens (candidate information, target information) except ADAEBDBECD. · The 12th token E refers to past tokens (candidate information, target information, C) except ADAEBDBECD.

[0225] When generating output information for three tokens, the machine learning model M performs one inference process using information in which N_1*N_2*2 tokens are added to the end of the target information, where N_1 is the number of options in the first classification and N_2 is the number of options in the second classification.

[0226] In general, when generating output information for K tokens, the machine learning model M performs one inference process using information with N_1*···*N_K-1*(K-1) tokens added to the end of the target information, where K is an integer greater than or equal to 2 and N_K-1 is the number of options for the K-1th classification.

[0227] Although an example of answering a question by selecting options has been described here, this embodiment can be applied to any task that generates output information that can be expressed by two or more tokens. An example of information that can be expressed by two or more tokens is numbers from 0000 to 0000. Therefore, this embodiment is not limited to tasks that involve answering by classification or selecting options, and can be applied to any task.

[0228] <User Interface> 17 is a diagram showing an example of an input screen according to the fourth embodiment. Here, differences from the input screen 400 (see FIG. 6) according to the first embodiment will be mainly described.

[0229] As shown in FIG. 17, the input screen 450 may have multiple question input sections 401 (401-1, 401-2), multiple option input sections 402 (402-1, 402-2), multiple option addition buttons 403 (403-1, 403-2), an input information display section 404, a target information input section 405, and an add question button 451.

[0230] Add question button 451 is a button for adding a new question and options to input screen 450. When the user presses add question button 451, question input section 401 and option input section 402 are added to input screen 450. Note that although the same question is input in option input section 402-1 and option input section 402-2, different questions may be input instead.

[0231] The input information display section 404 displays model input information including a plurality of questions and a plurality of options input to the question input section 401 and the option input section 402. The model input information displayed in the input information display section 404 includes a question and option set 452-1 and a question and option set 452-2. The question and option set 452-1 includes a question and options corresponding to the question input section 401-1 and the option input section 402-1. The question and option set 452-2 includes a question and options corresponding to the question input section 401-2 and the option input section 402-2.

[0232] The input information display unit 404 may automatically generate model input information based on the information input to the question input units 401-1 and 401-2 and the option input units 402-1 and 402-2. The input screen 450 may have a button for generating model input information to be displayed on the input information display unit 404. The input information display unit 404 may display the model input information in a format that can be edited by the user using a text editor or the like.

[0233] When candidate information including multiple questions and options is included in an inference request, output information including answers to each question may be generated in a single inference process by the inference process according to this embodiment. On the input screen 450, the reference answer information 413 included in the input information display section 404 may include answers that can be expressed with two tokens.

[0234] In this embodiment, for each of the multiple questions input to question input units 401-1 and 401-2, an inference result similar to that of output screen 500 according to the first embodiment (see FIG. 7) may be displayed. That is, output screen 500 according to this embodiment may display the probability value for each option for each of the multiple questions, may display the option with the highest probability value, or may display the option with the highest probability value and the probability value of that option. Furthermore, output screen 500 according to this embodiment may display options for each of the multiple questions in descending or ascending order of probability value.

[0235] [Another embodiment 1] In the above embodiment, numbers, letters, symbols, etc. are used as identification information for options, but special tokens may also be used as identification information for options. A special token is a token that is not linked to a specific character string. For example, a special token is: <choice1> , <choice2>,···, etc. may also be used.

[0236] To enable the use of special tokens, the machine learning model M may perform pre-training by replacing training data including options with special tokens. In an inference process using the machine learning model M, when generating model input information, identification information of options included in candidate information may be replaced with special tokens. By using special tokens as identification information of options, it is possible to generate output information that is free from the influence of biases inherent in numbers, letters, symbols, etc. used in the identification information.

[0237] In the above embodiment, a configuration has been described in which intermediate data is calculated by inputting candidate information into a machine learning model, and a predetermined number of pieces of target information are batch-processed using the intermediate data corresponding to the candidate information. Alternatively, a piece of target information may be input into the machine learning model together with the candidate information to execute inference processing, thereby generating output information for the target information together with the intermediate data corresponding to the candidate information.

[0238] Specifically, batch processing may be performed with the following configuration. In the first batch processing, the batch size is set to 1, and candidate information and one piece of target information are input to the machine learning model. The machine learning model outputs intermediate data corresponding to the candidate information and output information for the target information. In the second and subsequent batch processing, the batch size is set to a larger value, and target information equal to the batch size is batch-processed using the intermediate data calculated in the first batch processing. After that, batch processing is repeatedly performed with larger batch sizes until output information is generated for all of the target information.

[0239] In the above embodiment, as an example, before executing an inference process for each of the plurality of pieces of target information, candidate information is input into the machine learning model M, an inference process is executed, and intermediate data is cached, and the cached intermediate data is used to execute the inference process for each of the plurality of pieces of target information. In another embodiment, the second inference unit 130 may combine at least a portion of the candidate information with the target information and input the combined information as input information to the machine learning model M, thereby executing the inference process for the target information. In this case, the information processing system 1000 may not need to calculate intermediate data using the candidate information in advance. In other words, the information processing system 1000 may be configured without the functions of the first inference unit 120 and the state storage unit 102.

[0240] As an example, the second inference unit 130 may generate output information by inputting the following prompt, which combines the candidate information 310 in Figure 3 and the target information ("I went to this store and it was terrible"), into the machine learning model M. ·prompt: Please rate your impressions of the following statements.

[0241] Select from: 1: Good impression 2: Bad impression 3: Neither Text: I went to this store and it was awful

[0242] This prompt includes information (at least a portion of the candidate information) requesting the generation of information (e.g., one token) that can be generated in a single inference process using the machine learning model M. Therefore, the information processing system 1000 can generate probability information for each option by executing a single inference process. Here, the second inference unit 130 can omit execution of subsequent inference processes, eliminating the need to cache intermediate data (key vectors, value vectors) calculated during the execution of a single inference process for subsequent inference processes. The information processing system 1000 may use a similar technique to batch-process each of multiple pieces of target information combined with at least a portion of the candidate information. Because the information processing system 1000 can omit caching of the candidate information and the target information, it can generate output information for each of multiple pieces of input information with minimal memory usage. The information processing system 1000 may include reference answer information in the input information (prompt), as in the above embodiment.

[0243] Note that if the target information itself includes information that requires the generation of information (for example, one token) that can be generated in one inference process using the machine learning model M, it is not necessary to perform the process of combining with at least a portion of the candidate information. Examples of input information (prompts) in this case include the following: Example 1: "Please answer the number of days in a week."

[0244] The input information in this embodiment may be information combining at least a portion of the candidate information with the target information, or may be target information including information requesting the generation of information (e.g., one token) that can be generated in a single inference process using the machine learning model M. The input information may further include reference answer information. The information processing system 1000 may execute a single inference process for each of the multiple pieces of input information. In this case, the inference processes may be executed in parallel using batch processing. The information processing system 1000 may be configured not to execute subsequent inference processes for each of the multiple pieces of input information. The information processing system 1000 may be configured to execute only a single inference process for each of the multiple pieces of input information. The information processing system 1000 may not cache intermediate data calculated in the execution of the inference process for each of the multiple pieces of input information. The input information may include text data, image data, or audio data.

[0245] In the above embodiment, as an example, output information for each piece of input information (including target information and reference information) is generated by executing one inference process for each piece of input information. Here, one inference process may be one output information generation process included in a process of inputting input information into a machine learning model and autoregressively generating output information (tokens, etc.) multiple times. As a non-limiting example, one inference process may be one token generation process.

[0246] In the above embodiment, an example has been described in which the information processing system 1000 executes a task of determining the degree of positive or negative impression of a message posted on a social networking service. However, the tasks executed by the information processing system 1000 are not limited to the above.

[0247] Other examples of generation tasks include a task of extracting information related to a specific topic from news articles, a task of analyzing whether posts about a specific product on a social networking service are favorable, a task of extracting dissatisfied reviews posted about a specific product on a mail-order service, a task of filtering reviews posted about a specific product that relate to a specific defect, a task of filtering training data for machine learning models such as foundation models, marketing analysis of user reputation or feedback from users, a task of extracting necessary data from big data, and a task of removing noise from big data.

[0248] In each of the above embodiments, a configuration has been described in which the inference device 10 accepts input of candidate information and target information via a user interface and presents an inference result for the target information to a user. The inference device 10 may provide input of at least one of candidate information and target information, output of output information for the target information, etc., via an API (Application Programming Interface). The API may be a Web API provided via the Web.

[0249] In the above embodiments, a configuration has been described in which a decoder-only large-scale language model is used as an example of the machine learning model M. However, the machine learning model M is not limited to a decoder-only large-scale language model. The machine learning model M may be any machine learning model that can reduce calculation costs by using a cache.

[0250] The information processing system 1000 may periodically execute the same task repeatedly. As an example, the information processing system 1000 may execute the same task multiple times on the same processing target at different times. The information processing system 1000 may execute the same task once for different processing targets. In this case, the candidate information held by the inference device 10 may be used without acquiring candidate information from the terminal device 50 each time a task is executed.

[0251] The information processing system 1000 can classify or analyze large amounts of data. As an example, the information processing system 1000 may generate a prompt including an image modal and generate output information that evaluates whether the prompt is similar or dissimilar to the target information. The information processing system 1000 may also filter the target information based on the output information. With the above configuration, classification or analysis of large amounts of data can be performed without additional training of a machine learning model. In other words, the information processing system 1000 can classify or filter large amounts of data without prior training by utilizing knowledge already learned by the machine learning model.

[0252] The information processing system 1000 can build various classifiers simply by changing the prompts. As an example, the information processing system 1000 may perform class classification using few-shot prompting.

[0253] All configurations related to the information processing system 1000 according to each of the above-described embodiments can be applied to the information processing system 1000 according to the other embodiments. As an example, the configurations related to the information processing system 1000 according to the second to fourth embodiments can be applied to the information processing system 1000 according to the first embodiment. The configurations related to the information processing system 1000 according to the third or fourth embodiment can be applied to the information processing system 1000 according to the second embodiment. The configurations related to the information processing system 1000 according to the fourth embodiment can be applied to the information processing system 1000 according to the third embodiment.

[0254] <Summary> As is clear from the above explanation, the information processing system 1000 according to one embodiment of the present disclosure acquires information regarding output candidates and multiple pieces of target information, calculates first intermediate data by inputting the information regarding the output candidates into a machine learning model, and generates output information for each of the multiple pieces of target information by using at least a portion of the intermediate data to perform a single inference process using the machine learning model for each of the multiple pieces of target information.

[0255] The information about the output candidates may include identification information of the options. The identification information of the options may be information that can be represented using information that can be generated in a single inference process. The information about the output candidates may include options. The options may be information that can be represented using information that can be generated in a single inference process. The information about the output candidates may include information requesting the generation of numbers. The numbers may be information that can be represented using information that can be generated in a single inference process. The information about the output candidates may include information requesting the generation of information that can be generated in a single inference process.

[0256] The information processing system 1000 may not store at least a portion of the second intermediate data calculated by executing a single inference process on the target information for use in subsequent inference processes on the target information using the machine learning model. Not storing at least a portion of the second intermediate data may include at least one processor not caching at least a portion of the second intermediate data. At least a portion of the second intermediate data may include at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model.

[0257] The information processing system 1000 may cache at least a portion of the first intermediate data and execute a single inference process using a machine learning model for each of the plurality of target information using at least a portion of the cached first intermediate data. At least a portion of the first intermediate data may include at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model.

[0258] The information processing system 1000 may not execute any inference processing after the first inference processing for the target information. The information processing system 1000 may generate output information for each of two or more pieces of target information in parallel.

[0259] The information about the output candidates may include at least information about the first classification and information about the second classification. The information processing system 1000 may input information combining the target information and an option token of the first classification to the machine learning model in one inference process. The option token may be combined after the target information. The option token may include at least a first option token related to the first classification and a second option token related to the first classification that is combined after the first option token. In one inference process, the second option token may be set not to refer to the first option token.

[0260] The information processing system 1000 may train another model based on the output information and the correct answer information for each of the plurality of pieces of target information. The information processing system 1000 may generate second output information for each of the plurality of pieces of target information by inputting the output information for each of the plurality of pieces of target information to the other model.

[0261] The information processing system 1000 may select one or more pieces of target information to be used in generating third output information based on output information for each of the multiple pieces of target information. The information processing system 1000 may generate input information to be input to at least one of the machine learning model or another machine learning model based on the generation request and the one or more pieces of target information. The information processing system 1000 may generate the third output information by inputting the input information to at least one of the machine learning model or another machine learning model. The information regarding the output candidates may be information generated based on the generation request. The multiple pieces of target information may be information acquired by a search based on specified search conditions. The specified search conditions may be determined based on the generation request.

[0262] The terminal device 50 may display, on the display device, information for acquiring information about output candidates, information for acquiring a plurality of pieces of target information, and an inference result based on the output information for each piece of target information. The output information for each piece of target information may be information generated by executing a single inference process using the machine learning model for each piece of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model.

[0263] The terminal device 50 may acquire information about options from a user and generate information about output candidates based on the information about the options. The information about the options may include at least identification information of options that can be represented using information that can be generated in a single inference process, or options that can be represented using information that can be generated in a single inference process. The inference result may include at least a probability value of the option, an option selected based on the probability information included in the output information, or information based on the selected option. The terminal device 50 may acquire multiple pieces of target information from an electronic file specified by the user.

[0264] The information processing system 1000 acquires multiple pieces of input information, and generates output information for each of the multiple pieces of input information by performing a single inference process using a machine learning model on each piece of input information, and does not store at least a portion of the intermediate data calculated by performing a single inference process on the input information for inference processes subsequent to the single inference process using the machine learning model on the input information, and the input information may include information requesting the generation of information that can be generated in a single inference process.

[0265] Not storing at least a portion of the intermediate data may include at least one processor not caching at least a portion of the intermediate data. At least a portion of the intermediate data may include at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model. The information processing system 1000 may not perform inference processing on the input information after one inference processing.

[0266] As a result, according to one embodiment of the present disclosure, a technique for generating output information for each of a plurality of pieces of information with few computational resources can be provided. In one aspect, according to one embodiment, output information for each of a plurality of pieces of target information can be generated with a small amount of computation. In another aspect, according to one embodiment, output information for each of a plurality of pieces of target information can be generated with a small amount of memory usage. As one example, according to this embodiment, a plurality of pieces of information can be classified with few computational resources. As another example, according to one embodiment, output information can be generated based on appropriate reference information in a search expansion generation framework.

[0267] [Hardware configuration of information processing device] Some or all of the devices (inference device 10, generation device 20, search device 30, and terminal device 50) in the above-described embodiments may be configured as hardware, or may be configured as software (program) information processing executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or the like. When configured as software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communication network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), thereby allowing the software information processing to be executed by hardware.

[0268] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk, memory, etc. The storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or outside the computer.

[0269] 18 is a block diagram showing an example of the hardware configuration of each device (inference device 10, generation device 20, search device 30, and terminal device 50) in the above-mentioned embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76.

[0270] Although the computer 7 in FIG. 18 includes one of each component, it may also include multiple of the same component. Also, while FIG. 18 shows one computer 7, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, a distributed computing configuration may be used in which each computer communicates via a network interface 74 or the like to execute the processing. That is, each device in the above-described embodiment (the inference device 10, the generation device 20, the search device 30, and the terminal device 50) may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize its functions. Furthermore, the system may be configured such that information sent from a terminal is processed by one or more computers provided on a cloud, and the processing results are sent to the terminal.

[0271] The various calculations of each device in the above-described embodiments (the inference device 10, the generation device 20, the search device 30, and the terminal device 50) may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. of the present disclosure may be realized by at least one of a processor and a storage device provided on a cloud that can communicate with the computer 7 via a network. Thus, each device in the above-described embodiments may be implemented in the form of parallel computing using one or more computers.

[0272] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls a computer or performs calculations. The processor 71 may be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. The processor 71 may also include an optical circuit or a calculation function based on quantum computing.

[0273] The processor 71 may perform arithmetic processing based on data or software input from each device or the like configured inside the computer 7, and may output the calculation results or control signals to each device or the like. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.

[0274] Each device in the above-described embodiment (inference device 10, generation device 20, search device 30, and terminal device 50) may be realized by one or more processors 71. Here, processor 71 may refer to one or more electronic circuits arranged on a single chip, or to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wire or wirelessly.

[0275] The main memory device 72 may store instructions executed by the processor 71, various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory devices for saving various data, etc. in each device (the inference device 10, the generation device 20, the search device 30, and the terminal device 50) in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by an internal memory built into the processor 71. For example, each memory unit in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73.

[0276] When each device in the above-described embodiment (inference device 10, generation device 20, search device 30, and terminal device 50) is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Also, a configuration in which at least one processor out of multiple processors is connected to at least one storage device out of multiple storage devices may be included. Also, this configuration may be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0277] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wire. The network interface 74 may be an appropriate interface, such as one that conforms to an existing communication standard. Information may be exchanged with an external device 9A connected via the communication network 8 via the network interface 74. The communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0278] The device interface 75 is an interface such as a USB that directly connects to the external device 9B.

[0279] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device connected directly to the computer 7.

[0280] For example, the external device 9A or the external device 9B may be an input device. The input device may be a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device equipped with an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0281] Furthermore, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the output device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0282] Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

[0283] Furthermore, external device 9A or external device 9B may be a device having some of the functions of the components of each device (inference device 10, generation device 20, search device 30, and terminal device 50) in the above-described embodiments. In other words, computer 7 may transmit some or all of the processing results to external device 9A or external device 9B, or may receive some or all of the processing results from external device 9A or external device 9B.

[0284] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0285] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data itself, unless otherwise specified. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of various data, etc.) is used as output, unless otherwise specified.

[0286] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0287] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure, etc. of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0288] Whenever words implying containing or possessing (e.g., "comprising / including," "having," etc.) are used in this specification (including the claims), they are intended to be open-ended terms that include the inclusion or possession of things other than the object designated by the object of the term. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

[0289] In this specification (including the claims), even if expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0290] In this specification, when a particular advantage / result is described as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or circumstances, and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or circumstances are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0291] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include electronic circuits, devices including electronic circuits, etc.

[0292] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0293] In this specification (including the claims), terms such as "first," "second," etc. are used merely as a way of distinguishing between two or more elements, and are not necessarily intended to impose technical meanings such as temporal aspect, spatial aspect, sequence, quantity, etc. Thus, for example, a reference to a first element and a second element does not necessarily mean that only two elements can be employed therein, that the first element must precede the second element, that the first element must be present in order for the second element to be present, etc.

[0294] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents. For example, when numerical values ​​or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure.

[0295] The disclosed technology may take the following forms as described below.

[0296] (Appendix 1) at least one memory; at least one processor; The at least one processor: Acquire information about output candidates and a plurality of pieces of target information; inputting information about the output candidates into a machine learning model to calculate first intermediate data; using at least a portion of the intermediate data to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information; Information processing system.

[0297] (Appendix 2) the information about the output candidates includes identification information of the options; The identification information of the option is information that can be expressed using information that can be generated in the single inference process. 10. The information processing system of claim 1.

[0298] (Appendix 3) the information about the output candidates includes options; The options are information that can be represented using information that can be generated in one inference process. 3. The information processing system according to claim 1 or 2.

[0299] (Appendix 4) the information about the output candidates includes information requesting the generation of a number; The number is information that can be expressed using information that can be generated in one inference process. 4. An information processing system according to any one of appendices 1 to 3.

[0300] (Appendix 5) the information about the output candidates includes information requesting generation of information that can be generated in one inference process; 5. An information processing system according to any one of appendices 1 to 4.

[0301] (Appendix 6) The at least one processor: not storing at least a portion of the second intermediate data calculated by executing the single inference process on the target information for use in inference processes subsequent to the single inference process on the target information using the machine learning model; 6. An information processing system according to any one of appendices 1 to 5.

[0302] (Appendix 7) not storing at least a portion of the second intermediate data includes the at least one processor not caching at least a portion of the second intermediate data. 7. The information processing system according to claim 6.

[0303] (Appendix 8) At least a portion of the second intermediate data includes at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model. 8. The information processing system according to claim 6 or 7.

[0304] (Appendix 9) The at least one processor: caching at least a portion of the first intermediate data; performing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of the cached first intermediate data; 9. An information processing system according to any one of appendices 1 to 8.

[0305] (Appendix 10) At least a portion of the first intermediate data includes at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model. 10. The information processing system of claim 9.

[0306] (Appendix 11) The at least one processor: No inference processing is performed on the target information after the first inference processing. 11. An information processing system according to any one of appendices 1 to 10.

[0307] (Appendix 12) The at least one processor: generating the output information for each of the two or more pieces of target information in parallel; 12. An information processing system according to any one of appendices 1 to 11.

[0308] (Appendix 13) the information about the output candidates includes at least information about a first classification and information about a second classification; The at least one processor: In the single inference process, information obtained by combining the target information and the tokens of the options in the first classification is input to the machine learning model; The tokens of the first category of options are combined after the target information, the first category of alternative tokens includes at least a first alternative token and a second alternative token that is combined after the first alternative token; In the single inference process, the token of the second option is set not to refer to the token of the first option. 13. An information processing system according to any one of appendices 1 to 12.

[0309] (Appendix 14) The at least one processor: learning another model based on the output information and correct answer information for each of the plurality of target information; 14. An information processing system according to any one of appendices 1 to 13.

[0310] (Appendix 15) The at least one processor: generating second output information for each of the plurality of pieces of target information by inputting the output information for each of the plurality of pieces of target information into another model; 15. An information processing system according to any one of appendices 1 to 14.

[0311] (Appendix 16) The at least one processor: selecting one or more pieces of target information to be used for generating third output information based on output information for each of the plurality of pieces of target information; generating input information to be input to at least one of the machine learning model and another machine learning model based on a generation request and the one or more pieces of target information; generating the third output information by inputting the input information to at least one of the machine learning model and the other machine learning model; 16. An information processing system according to any one of appendices 1 to 15.

[0312] (Appendix 17) the information about the output candidates is information generated based on the generation request; 17. The information processing system of claim 16.

[0313] (Appendix 18) the plurality of pieces of target information are pieces of information acquired by a search based on predetermined search conditions, the predetermined search conditions are determined based on the generation request; 18. The information processing system according to claim 16 or 17.

[0314] (Appendix 19) At least one memory; at least one processor; The at least one processor: Displaying information for acquiring information about output candidates on a display device; Displaying information for acquiring a plurality of pieces of target information on the display device; displaying an inference result based on output information for each of the plurality of pieces of target information on the display device; The output information for each of the plurality of pieces of target information is information generated by executing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model. Information processing device.

[0315] (Appendix 20) The at least one processor: Obtain information about your choices from the user; generating information about the output candidates based on information about the options; 19. The information processing device according to claim 18.

[0316] (Appendix 21) The information about the options includes at least identification information of options that can be represented using information that can be generated in the single inference process, or options that can be represented using information that can be generated in the single inference process. 21. The information processing device according to claim 20.

[0317] (Appendix 22) The inference result includes at least one of a probability value of the option, an option selected based on probability information included in the output information, or information based on the selected option. 22. The information processing device according to claim 20 or 21.

[0318] (Appendix 23) The at least one processor: acquiring the plurality of pieces of target information from an electronic file designated by a user; 23. An information processing device according to any one of appendices 19 to 22.

[0319] (Appendix 24) At least one memory; at least one processor; The at least one processor: Take multiple inputs, generating output information for each of the plurality of pieces of input information by executing a single inference process using a machine learning model for each of the plurality of pieces of input information; not storing at least a portion of the intermediate data calculated by executing the single inference process on the input information for use in subsequent inference processes on the input information after the single inference process using the machine learning model; the input information includes information requesting generation of information that can be generated in one inference process; Information processing system.

[0320] (Appendix 25) not storing at least a portion of the intermediate data includes the at least one processor not caching at least a portion of the intermediate data. 25. The information processing system of claim 24.

[0321] (Appendix 26) At least a portion of the intermediate data includes at least a portion of key vectors and value vectors used in an attention mechanism of the machine learning model. 26. An information processing system according to claim 24 or 25.

[0322] (Appendix 27) The at least one processor: Do not execute any inference processing subsequent to the first inference processing for the input information. 27. An information processing system according to any one of appendices 24 to 26.

[0323] (Appendix 28) At least one processor Acquire information about output candidates and a plurality of pieces of target information; inputting information about the output candidates into a machine learning model to calculate first intermediate data; using at least a portion of the intermediate data to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information; Information processing methods.

[0324] (Appendix 29) At least one processor Displaying information for acquiring information about output candidates on a display device; Displaying information for acquiring a plurality of pieces of target information on the display device; displaying an inference result based on output information for each of the plurality of pieces of target information on the display device; The output information for each of the plurality of pieces of target information is information generated by executing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model. Information processing methods.

[0325] (Appendix 30) At least one processor Take multiple inputs, generating output information for each of the plurality of pieces of input information by executing a single inference process using a machine learning model for each of the plurality of pieces of input information; not storing at least a portion of the intermediate data calculated by executing the single inference process on the input information for use in subsequent inference processes on the input information after the single inference process using the machine learning model; the input information includes information requesting generation of information that can be generated in one inference process; Information processing methods.

[0326] (Appendix 31) At least one processor has Acquire information about output candidates and a plurality of pieces of target information; inputting information about the output candidates into a machine learning model to calculate first intermediate data; using at least a portion of the intermediate data to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information; A program for executing a process.

[0327] (Appendix 32) At least one processor has Displaying information for acquiring information about output candidates on a display device; Displaying information for acquiring a plurality of pieces of target information on the display device; displaying an inference result based on output information for each of the plurality of pieces of target information on the display device; The output information for each of the plurality of pieces of target information is information generated by executing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model. A program for executing a process.

[0328] (Appendix 33) At least one processor has Take multiple inputs, generating output information for each of the plurality of pieces of input information by executing a single inference process using a machine learning model for each of the plurality of pieces of input information; not storing at least a portion of the intermediate data calculated by executing the single inference process on the input information for use in subsequent inference processes on the input information after the single inference process using the machine learning model; the input information includes information requesting generation of information that can be generated in one inference process; A program for executing a process. [Explanation of symbols]

[0329] 10: Reasoning device 20:Generation device 30: Search device 50: Terminal device 101: Model memory unit 102: State memory unit 110: Request reception unit 120: 1st reasoning part 130:Second reasoning part 140: Output section 210: Search section 220: Selection section 230: Generation part 1000: Information Processing Systems D: Data Source M: Machine learning model < / choice1>

Claims

1. at least one memory; at least one processor; The at least one processor Acquire information about output candidates and a plurality of pieces of target information; inputting information about the output candidates into a machine learning model to calculate first intermediate data; using at least a portion of the first intermediate data to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information; Information processing system.

2. the information about the output candidates includes identification information of the options; the identification information of the option is information that can be expressed using information that can be generated in one inference process; The information processing system according to claim 1 .

3. the information about the output candidates includes options; The options are information that can be represented using information that can be generated in one inference process. The information processing system according to claim 1 .

4. the information about the output candidates includes information requesting the generation of a number; The number is information that can be expressed using information that can be generated in one inference process. The information processing system according to claim 1 .

5. the information about the output candidates includes information requesting generation of information that can be generated in one inference process; The information processing system according to claim 1 .

6. The at least one processor at least a portion of second intermediate data calculated by executing the single inference process on the target information is not stored for inference processes subsequent to the single inference process on the target information using the machine learning model; The information processing system according to claim 1 .

7. not storing at least a portion of the second intermediate data includes the at least one processor not caching at least a portion of the second intermediate data. The information processing system according to claim 6.

8. At least a portion of the second intermediate data includes at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model. The information processing system according to claim 6.

9. The at least one processor caching at least a portion of the first intermediate data; performing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of the cached first intermediate data; The information processing system according to claim 1 .

10. At least a portion of the first intermediate data includes at least a portion of a key vector and a value vector used in an attention mechanism of the machine learning model. The information processing system according to claim 9 .

11. The at least one processor No inference processing is performed on the target information after the first inference processing.

11. The information processing system according to claim 1.

12. The at least one processor generating the output information for each of the two or more pieces of target information in parallel; 11. The information processing system according to claim 1.

13. the information about the output candidates includes at least information about a first classification and information about a second classification; The at least one processor In the single inference process, information combining the target information and the tokens of the options in the first classification is input to the machine learning model; The tokens of the first category of options are combined after the target information; the first category of alternative tokens includes at least a first alternative token and a second alternative token that is combined after the first alternative token; In the single inference process, the token of the second option is set not to refer to the token of the first option.

11. The information processing system according to claim 1.

14. The at least one processor learning another model based on the output information and correct answer information for each of the plurality of target information; 11. The information processing system according to claim 1.

15. The at least one processor generating second output information for each of the plurality of pieces of target information by inputting the output information for each of the plurality of pieces of target information into another model; 11. The information processing system according to claim 1.

16. The at least one processor selecting one or more pieces of target information to be used for generating third output information based on output information for each of the plurality of pieces of target information; generating input information to be input to at least one of the machine learning model and another machine learning model based on a generation request and the one or more pieces of target information; generating the third output information by inputting the input information to at least one of the machine learning model and the other machine learning model; 11. The information processing system according to claim 1.

17. the information about the output candidates is information generated based on the generation request; 17. The information processing system according to claim 16.

18. the plurality of pieces of target information are pieces of information acquired by a search based on predetermined search conditions, the predetermined search conditions are determined based on the generation request; 17. The information processing system according to claim 16.

19. at least one memory; at least one processor; The at least one processor Displaying information for acquiring information about output candidates on a display device; Displaying information for acquiring a plurality of pieces of target information on the display device; displaying an inference result based on output information for each of the plurality of pieces of target information on the display device; The output information for each of the plurality of pieces of target information is information generated by executing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model. Information processing device.

20. The at least one processor Obtain information about your choices from the user; the information about the output candidates is generated based on the information about the options. The information processing device according to claim 19.

21. the information about the options includes at least identification information of options that can be represented using information that can be generated in the single inference process, or options that can be represented using information that can be generated in the single inference process; The information processing device according to claim 20.

22. The at least one processor Get the choices from the user, assigning identification information to the options; the information about the output candidates is generated based on the options and the identification information; the identification information is information that can be expressed using information that can be generated in one inference process; The information processing device according to claim 19.

23. The inference result includes at least one of a probability value of the option, an option selected based on probability information included in the output information, or information based on the selected option.

23. The information processing device according to claim 20.

24. The at least one processor acquiring the plurality of pieces of target information from a designated electronic file; 23. The information processing device according to claim 19.

25. at least one memory; at least one processor; The at least one processor Take multiple inputs, generating output information for each of the plurality of pieces of input information by executing a single inference process using a machine learning model for each of the plurality of pieces of input information; not storing at least a portion of intermediate data calculated by executing the single inference process on the input information for use in subsequent inference processes on the input information after the single inference process using the machine learning model; the input information includes information requesting generation of information that can be generated in one inference process; Information processing system.

26. not storing at least a portion of the intermediate data includes the at least one processor not caching at least a portion of the intermediate data.

26. The information processing system according to claim 25.

27. At least a portion of the intermediate data includes at least a portion of key vectors and value vectors used in an attention mechanism of the machine learning model.

26. The information processing system according to claim 25.

28. The at least one processor No inference processing is performed after the first inference processing for the input information.

28. An information processing system according to any one of claims 25 to 27.

29. At least one processor Acquire information about output candidates and a plurality of pieces of target information; inputting information about the output candidates into a machine learning model to calculate first intermediate data; using at least a portion of the first intermediate data to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information; Information processing methods.

30. At least one processor Displaying information for acquiring information about output candidates on a display device; Displaying information for acquiring a plurality of pieces of target information on the display device; displaying an inference result based on output information for each of the plurality of pieces of target information on the display device; The output information for each of the plurality of pieces of target information is information generated by executing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model. Information processing methods.

31. At least one processor Take multiple inputs, generating output information for each of the plurality of pieces of input information by executing a single inference process using a machine learning model for each of the plurality of pieces of input information; not storing at least a portion of intermediate data calculated by executing the single inference process on the input information for use in subsequent inference processes on the input information after the single inference process using the machine learning model; the input information includes information requesting generation of information that can be generated in one inference process; Information processing methods.

32. At least one processor Acquire information about output candidates and a plurality of pieces of target information; inputting information about the output candidates into a machine learning model to calculate first intermediate data; using at least a portion of the first intermediate data to perform a single inference process using the machine learning model for each of the plurality of pieces of target information, thereby generating output information for each of the plurality of pieces of target information; A program for executing a process.

33. At least one processor Displaying information for acquiring information about output candidates on a display device; Displaying information for acquiring a plurality of pieces of target information on the display device; displaying an inference result based on output information for each of the plurality of pieces of target information on the display device; The output information for each of the plurality of pieces of target information is information generated by executing a single inference process using the machine learning model for each of the plurality of pieces of target information, using at least a portion of intermediate data calculated by inputting information about the output candidates into the machine learning model. A program for executing a process.

34. At least one processor Take multiple inputs, generating output information for each of the plurality of pieces of input information by executing a single inference process using a machine learning model for each of the plurality of pieces of input information; not storing at least a portion of intermediate data calculated by executing the single inference process on the input information for use in subsequent inference processes on the input information after the single inference process using the machine learning model; the input information includes information requesting generation of information that can be generated in one inference process; A program for executing a process.

Images